JP2017073710A - Element image group generation device and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an element image group by reducing influence caused by an error included in a three-dimensional model of a subject.SOLUTION: An element image group generation device 100 comprises: beam tracking means 110 for calculating coordinates of an intersection between a linear line showing a beam that is originated from a pixel of a virtual display disposed in a virtual space before passing through a lens of a virtual lens array and a three-dimensional model disposed in the virtual space; image integration means 120 that, on the basis of angles between camera vectors showing photographing directions by a plurality of cameras arranged in the virtual space on the basis of a camera parameter and a beam vector, a direction vector of the linear line, selects a camera having an angle equal to or lower than a predetermined value, projects the coordinates of the intersection on a camera image input with respect to the selected camera, and allocates pixel values of a point of the projection or a pixel value obtained by synthesizing the pixel values to the pixel of the virtual display; and image generation control means 130 for generating an element image group obtained by integrating pixel values allocated to all pixels of the virtual display.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an element image group generation device for stereoscopic television and a program thereof.

  A lens array in which microlenses are arranged in a two-dimensional array is placed in front of a two-dimensional display, and a predetermined image is displayed on the two-dimensional display, thereby emitting light beams having different luminance values (colors) depending on directions. Can do. When this is used, a three-dimensional image can be reproduced without glasses because different light rays are received by the right eye and the left eye depending on the position of the eyes. A stereoscopic television using this principle is called an integral stereoscopic television.

  A pixel group assigned to each microlens (element lens: hereinafter, also simply referred to as a lens) constituting the lens array is called an element image. An element image group in which the element images are accumulated is displayed on the two-dimensional display. In the integral method, the element image group is generally acquired by photographing a subject through a lens array using a high-definition camera having a large number of high-definition pixels. However, if a distant subject is photographed through a lens array, when this subject is displayed, it becomes a planar three-dimensional image with little stereoscopic effect.

  Therefore, instead of photographing the subject through the lens array, an element image group is generated from multi-viewpoint images acquired by a plurality of cameras, and an integral stereoscopic image is generated using the element image group generated by this method. Methods are conventionally known (see Non-Patent Document 1 and Non-Patent Document 2). In this method, first, a three-dimensional model (three-dimensional shape) of a subject is generated from a multi-view video using an existing three-dimensional shape restoration method. The three-dimensional model is arranged in the virtual space in the computing machine, and the virtual lens array and the virtual camera are also arranged in the virtual space in the same manner as in the real space. A three-dimensional image can be generated.

Yasutaka Furukawa and Jean Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, July 2007. K. Hisatomi, K. Tomiyama, M. Katayama, and Y. Iwadate, “3D archive system for traditional performing arts”, International Journal of Computer Vision (IJCV), Vol. 94, pp78-88 (2011)

However, the conventional technology assumes that a three-dimensional model of a subject is arranged in a virtual space, but the three-dimensional model itself has an error.
In addition, since the multi-view video used for generating the element image group is generated by photographing the same subject with a plurality of cameras, there is an overlapping portion in each camera image. Therefore, when generating an element image group from a three-dimensional model, it is very important which camera image (which camera) is selected for a certain pixel and whether the selected camera is appropriate.
Conventionally, in general, the camera closest to the normal line of the 3D model is selected, and when the integral stereoscopic image reproduced when displayed is viewed from a distance, the 3D model is selected. There was a problem that the error of () was exaggerated. The angle formed by the direction in which the image was taken (the direction of the light beam emitted from the subject at the time of shooting and recorded in the camera) and the direction to be observed (the direction of the light beam that is reproduced when the subject is displayed on the two-dimensional display) As the value of becomes larger, the influence of errors in the three-dimensional model becomes larger.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide an element image group generation device and a program thereof that can reduce the influence of errors included in a three-dimensional model of a subject. To do.

  In order to solve the above-described problem, an element image group generation device according to a first aspect of the present invention uses a multi-view camera that captures a subject using a multi-view camera using a pre-generated three-dimensional model of the subject. An element image group generation apparatus that generates an element image group from an image, and includes a light ray tracing unit, an image integration unit, and an image generation control unit.

According to this configuration, the element image group generation device uses the ray tracing means to obtain a ray vector that is a linear direction vector representing rays that are emitted from the pixels of the virtual display arranged in the virtual space and pass through the lenses of the virtual lens array. The coordinates of the intersection determined corresponding to the position closest to the observer among the intersections of the straight line and the three-dimensional model arranged in the virtual space are obtained.
Then, the element image group generation device uses the image integration unit to generate a camera vector indicating the shooting direction of the plurality of cameras arranged in the virtual space based on the camera parameters of the multi-viewpoint camera used for shooting the subject and the light beam. Select a camera whose angle with the vector is equal to or less than a predetermined value, project the coordinates of the intersection point on the camera image input for the selected camera, and synthesize the pixel value of this projection point or them Assign pixel values to the pixels of the virtual display.
This makes it possible to select a camera photographed from the direction closest to the direction of the light beam observed when the subject is displayed from among a plurality of cameras that photographed the subject.
Then, the element image group generation device generates an element image group in which pixel values assigned to all the pixels of the virtual display are accumulated by the image generation control unit.
Therefore, the element image group generation device generates a stereoscopic image using the video of the camera taken from the direction closest to the direction of the light beam to be observed. Therefore, the effect of the error in the three-dimensional model arranged in the virtual space in the computing machine is reduced. Can be minimized.

  In order to solve the above problem, an element image group generation device according to a second aspect of the present invention uses a plurality of depth maps generated in advance of a subject, and uses a multi-view camera to capture the subject. An element image group generation apparatus that generates an element image group from a viewpoint camera image, and includes a ray tracing unit, an image integration unit, and an image generation control unit.

According to this configuration, the element image group generation device uses the ray tracing means to obtain a ray vector that is a linear direction vector representing rays that are emitted from the pixels of the virtual display arranged in the virtual space and pass through the lenses of the virtual lens array. The coordinates of the intersection determined corresponding to the position closest to the observer among the intersections of the straight line and the depth map arranged in the virtual space are obtained.
Then, the element image group generation device uses the image integration unit to generate a camera vector indicating the shooting direction of the plurality of cameras arranged in the virtual space based on the camera parameters of the multi-viewpoint camera used for shooting the subject and the light beam. Select a camera whose angle with the vector is equal to or less than a predetermined value, project the coordinates of the intersection point on the camera image input for the selected camera, and synthesize the pixel value of this projection point or them Assign pixel values to the pixels of the virtual display.
Then, the element image group generation device generates an element image group in which pixel values assigned to all the pixels of the virtual display are accumulated by the image generation control unit.

  The present invention can also be realized by an element image group generation program for causing a computer to function as the element image group generation apparatus.

  According to the present invention, it is possible to generate an element image group by reducing the influence of errors included in a three-dimensional model of a subject.

It is a block diagram which shows typically the structure of the element image group production | generation apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram which shows an example of arrangement | positioning of the multiview camera which image | photographed each camera image input into the element image group generation apparatus of FIG. It is explanatory drawing of operation | movement of the element image group production | generation apparatus of FIG. 1, Comprising: The virtual space to set is shown typically. It is explanatory drawing of operation | movement of the element image group production | generation apparatus of FIG. 1, Comprising: The light vector is typically shown in virtual space. FIGS. 2A and 2B are explanatory diagrams of the operation of the element image group generation device of FIG. 1, in which FIG. 1A shows an example of pixel value assignment processing, and FIG. It is a flowchart which shows the flow of the process by the element image group production | generation apparatus of FIG. It is a block diagram which shows typically the structure of the element image group production | generation apparatus which concerns on 2nd Embodiment of this invention. FIG. 8 is an explanatory diagram of the operation of the element image group generation device in FIG. 7, schematically showing a virtual space and a light vector. It is a flowchart which shows the flow of a process by the element image group generation apparatus of FIG.

An embodiment for implementing an element image group generation device according to the present invention will be described in detail with reference to the drawings.
(First embodiment)
[Configuration of Element Image Group Generation Device]
An element image group generation device 100 shown in FIG. 1 generates a stereoscopic image of an integral 3D television from a multi-view camera image (multi-view video) obtained by shooting the same subject with a multi-view camera (a plurality of cameras). An element image group is generated. The element image group generation device 100 according to the present embodiment generates an element image group by arranging a previously generated three-dimensional model of a subject in a virtual space in a computing machine.

  As shown in FIG. 1, the element image group generation apparatus 100 mainly includes a ray tracing unit 110, an image integration unit 120, and an image generation control unit 130. The element image group generation device 100 is configured by, for example, a general computer, and includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), an input / output interface, and the like. It has. Note that the virtual space information memory 114 and the element image group storage memory 131 are configured by an HDD or a general image memory.

  In the present embodiment, a camera image obtained by photographing a subject from outside during the processing of the element image group generation device 100, camera parameters acquired in advance, display parameters and lens array parameters set as appropriate, and the subject generated in advance. It is assumed that a three-dimensional model is input. These input data may be stored in the internal memory of the element image group generation device 100 before processing.

First, input data will be described.
<Input data>
A camera image represents at least one viewpoint video of a multi-view video shot by a multi-view camera. It should be noted that, regarding viewpoint video and multi-view video, when describing image processing, or when referring to a stored image or an image in units of frames, it may be referred to as a camera image or simply an image. An example of the arrangement of multi-viewpoint cameras used for shooting is shown in FIG. In the example shown in FIG. 2, a two-dimensional camera array using a total of 21 color cameras C (hereinafter simply referred to as cameras) of 3 × 7 in the vertical direction is used. However, the number of cameras is not limited to this, and may be a one-dimensional camera array including a total of seven cameras C, for example, 1 × 7 vertically.

The camera parameters include lens system parameters such as the focal length of each camera for the multi-viewpoint camera used for photographing the subject, and parameters relating to postures such as rotation and translation of each camera.
As a method for acquiring camera parameters, for example, after a multi-viewpoint camera is installed and before a subject is photographed, a pattern in which a point cloud is drawn at a known position is simultaneously photographed and photographed by all cameras. It is possible to acquire camera parameters by using a technique as described in Reference Document 1 below using all camera images.
Reference 1: R.R. Y. Tsai, “A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses”, IEEE Journal of Robotics and Automation, RA-3 (4), pp323-344, 1987

Alternatively, instead of using a pattern in which a point cloud is drawn at a known position, for example, a feature point extracted from the image photographed by the camera by using a technique as described in Reference Document 2 below. It is also possible to acquire camera parameters using the correspondence between the images.
Reference 2: Noah Snavely, Steven M. et al. Seitz, Richard Szeliski, “Modeling the World from Internet Photo Collections”, International Journal of Computer Vision, 80 (2), 189-210, November 2008

Examples of the display parameter include the vertical and horizontal sizes of the virtual display, the size of the pixels on the virtual display, the number of pixels, the pixel pitch, and the position of each pixel.
The lens array parameters include, for example, the vertical and horizontal sizes of the virtual lens array, the diameter and number of lenses (microlenses) constituting the virtual lens array, the lens pitch, the position of the optical principal point of each lens, the focal length of the lens, and the like. Can be mentioned.
If the display or lens array in which the image is to be displayed is known, the parameters of the display or lens array may be applied as they are.

The three-dimensional model represents the three-dimensional shape of the subject and is generated in advance. In the present embodiment, the method for generating the three-dimensional model is not particularly limited.
As an example, a description will be given assuming that a subject is restored from a multi-view video obtained by shooting with a multi-view camera. As a method for generating a three-dimensional model from a multi-view video, a stereo method for integrating a model generated sequentially from two camera pairs with a multi-view method that generates a model at once using all camera images. A visual method is known. A visual volume intersection method is also known as one of the multi-view methods. Any of these methods can be employed.

Next, each unit constituting the element image group generation device 100 will be described.
<Ray tracing means>
The ray tracing unit 110 tracks rays emitted from the pixels of the virtual display arranged in the virtual space under the control of the image generation control unit 130.
The ray tracing unit 110 receives display parameters, lens array parameters, camera parameters, and a three-dimensional model. The ray tracing unit 110 includes a virtual space setting unit 111, a ray vector calculation unit 112, and a position information calculation unit 113.

First, the virtual space setting unit 111 will be described with reference to FIG. 3 (refer to FIG. 1 as appropriate).
The virtual space setting unit 111 arranges the virtual display at a reference position in the virtual space (virtual space information memory 114) based on the input display parameter.
In FIG. 3, as an example, the Y axis is arranged along the horizontal direction (horizontal direction) of the display screen (pixel group) of the virtual display, and the vertical direction (direction perpendicular to the paper surface) of the display screen (pixel group) of the virtual display. ) Along the X axis.

  The virtual space setting unit 111 arranges the virtual lens array in parallel with the display screen of the virtual display in the virtual space (virtual space information memory 114) based on the input lens array parameters. Here, the vector from the center of the virtual display to the center of the virtual lens array is taken as the Z axis. The virtual space setting unit 111 arranges the virtual lens array at a position separated from the virtual display by the focal length f in the Z-axis direction. In FIG. 3, the Y axis is shifted in the negative direction of the Z axis for easy viewing.

The virtual space setting unit 111 arranges a virtual camera (hereinafter also simply referred to as a camera) in the virtual space (virtual space information memory 114) based on the input camera parameters.
Hereinafter, as an example, it is assumed that a multi-view video of a subject is captured by a multi-view camera including five cameras. The position where each camera is arranged represents the viewpoint position. In this case, the multi-view video is composed of images (camera images) I ₀ , I ₁ , I ₂ , I ₃ , and I ₄ .

The virtual space setting unit 111 arranges the three-dimensional model Q in the vicinity of the virtual display in the virtual space (virtual space information memory 114).
Here, a three-dimensional model having a complicated shape is illustrated for the purpose of explaining an intersection with a straight line to be described later. The three-dimensional model Q shown in FIG. 3 has two projections that project from the virtual display to the virtual lens array side (observer side) in a plan view, and the virtual display to the virtual lens between the two projections. Consists of a curve with a recess recessed on the opposite side. In FIG. 3, the front (right side in FIG. 3) and side surfaces (upper and lower in FIG. 3) of the model are shown by a series of smooth curves, but in some cases, waves with fine zigzags It may be required as a shape model.

Next, referring to FIG. 4 (refer to FIG. 1 as appropriate), the description of the configuration of the ray tracing means 110 will be continued.
The light ray vector calculating means 112 calculates a light ray vector g which is a direction vector of a straight line L representing a light ray emitted from the pixel i of the virtual display arranged in the virtual space and passing through the lenses of the virtual lens array. Note that i is an identifier for specifying two-dimensional coordinates indicating the number of pixels from the left and the number of pixels from the top on the virtual display, and is represented by 0, 1,. The calculated light vector is output to the image integration unit 120.
In the present embodiment, the ray vector g is a vector whose end point is the position of the pixel i on the virtual display, and starts from the optical principal point p of the microlens m nearest to the pixel i in the virtual lens array. It was assumed to be a vector. Assuming a vector opposite to the direction of the ray vector g, the direction of the vector (the direction of the straight line L toward the virtual camera side) is the integrator reproduced from the pixel when the generated stereoscopic image is observed. Represents the direction of the rays of the 3D image.

The position information calculation means 113 calculates the coordinates of the intersection determined corresponding to the position closest to the observer among the intersections of the straight line L passing through the pixel i of the virtual display and the three-dimensional model Q arranged in the virtual space. Ask. In the example shown in FIG. 4, the position information calculation unit 113, among the intersection points _{{S 0, S 1, S} 2} between the straight line L and the three-dimensional model Q through the pixel i, most Z value is large intersection S ₂ The coordinates are obtained and output to the image integration unit 120. In the following description, the intersection S is simply referred to unless otherwise distinguished.

<Image integration means>
The image integration unit 120 integrates the pixel values using the camera images under the control of the image generation control unit 130. The image integration unit 120 receives a camera image and camera parameters. As shown in FIG. 1, the image integration unit 120 includes a camera vector calculation unit 121, a camera selection unit 122, and a pixel value assignment unit 123.

The camera vector calculation unit 121 calculates camera vectors indicating the shooting directions of a plurality of cameras arranged in the virtual space based on input camera parameters. The calculated camera vector is output to the camera selection unit 122.
3 and 4, the camera vector C ₀ indicates the shooting direction of the camera that shot the image I ₀ . Similarly, the other camera vectors C ₁ , C ₂ , C ₃ , and C ₄ indicate the shooting directions of the cameras that have shot the images I ₁ , I ₂ , I ₃ , and I ₄ , respectively. When the camera vector is not particularly distinguished, it is expressed as C _n . Also, the camera vector code (C _n ) may be used to identify the camera. When it can be easily distinguished from the camera itself, it is also expressed as a camera vector C. Incidentally, unless otherwise specified camera image, denoted as I _n or simply I.

The camera selection means 122 receives the light vector g and the camera vector C _n .
The camera selection unit 122 selects a camera in which an angle formed by the camera vector C _n and the light vector g in the virtual space is a predetermined value or less. That is, the camera selection unit 122 calculates the inner product of each camera vector C _n and the light vector g, and selects a camera whose inner product is _equal to or greater than a predetermined threshold value. Here, the predetermined value for the angle and the threshold value for the inner product are not particularly limited, and the camera selection unit 122 can select at least one camera according to the purpose and application of the element image group generation device 100. The value can be set as appropriate.

In the example shown in FIG. 4, it is assumed that the inner product of the light vector g and the camera vector C ₂ and the inner product of the light vector g and the camera vector C ₃ are equal to or greater than a predetermined threshold. In this case, the camera selection unit 122 selects the camera that has captured the image I ₂ and the camera that has captured the image I ₃ . Here, the camera is selected in order to select the camera image, and since the camera vector code (C _n ) also serves as the camera identifier, the camera vector {C ₂ , C ₃ } or a camera image {I ₂ , I ₃ }.

The camera selection means 122 selects a camera on the basis of the light vector g and the camera vector C _n. As described above, when the generated stereoscopic image is observed, the direction of the vector opposite to the direction of the ray vector g (the direction of the straight line L toward the virtual camera) is reproduced from the pixel. Represents the direction of rays of the integral 3D image. Therefore, by selecting a camera vector (camera) in the photographing direction close to the direction of the light vector g by the camera selection unit 122 and adopting the camera image I, the element image group generation device 100 can reproduce the integral to be reproduced. It is possible to minimize the influence of the error of the three-dimensional model Q in the stereoscopic image.

  The pixel value assigning means 123 assigns the pixel value of the projection point J obtained by projecting the intersection of the three-dimensional model Q onto the camera image I or the synthesized pixel value to the pixel of the virtual display. The pixel value assigning unit 123 includes a projecting unit 124 and a synthesizing unit 125.

The projection unit 124 projects the coordinates of the intersection point on the camera image input for the camera selected by the camera selection unit 122.
The projection unit 124 is provided on an imaging surface of a certain virtual camera arranged with a certain photographing direction with respect to a three-dimensional position of a certain point in the virtual space by calibration performed in advance on the virtual camera based on camera parameters. It is configured to be associated with the position of one point. The position association is performed by a calculation using a lookup table or a relational expression obtained based on calibration. Therefore, when the coordinates of the three-dimensional position in the virtual space (coordinates of the intersection) are input, the projection unit 124 can specify the position of the pixel of the camera image as the projection destination. For this calibration, for example, camera parameters obtained by simultaneously capturing a pattern in which a point cloud is drawn at a known position with a multi-view camera can be used.

  The synthesizing unit 125 synthesizes the pixel values of the projection points of the camera images when a plurality of cameras are selected by the camera selection unit 122. The combined pixel value V is output to the image generation control unit 130. Note that when only one camera is selected by the camera selection unit 122, the synthesis unit 125 outputs the pixel value of the projection point of the camera image to the image generation control unit 130 as it is.

Here, a specific example of combining pixel values will be described with reference to FIG. 5A (refer to FIGS. 1 and 4 as appropriate). For example, as a result of obtaining the inner product of each camera vector C _n and the light vector g passing through the pixel i in the camera selection means 122, two camera vectors {C ₂ , C ₃ } are selected as having an inner product _equal to or greater than a threshold value. Shall be. In this case, the projection unit 124 calculates the intersection S ₂ calculated by the position information calculation unit 113 for the _two camera images {I ₂ , I ₃ } corresponding to these camera vectors {C ₂ , C ₃ }. Project each coordinate. In the example shown in FIG. 5 (a), J ₂ the projection point of intersection S ₂ in the camera image I _2, and the projection point of intersection S ₂ in the camera image I ₃ as the J _3. Further, the pixel value of the projection point J ₂ (color value) V _2, the pixel value of the projection point J ₃ (color value) and V _3.

In this case, the combining unit 125 outputs the pixel value V as a result of combining the pixel value V ₂ and the pixel value V ₃ to the image generation control unit 130. Here, a plurality of pixel values (color values) can be combined by an expression defined by an angle formed by each camera vector {C ₂ , C ₃ } and the light vector g. For example, as shown in the following equation (1), the pixel value V is obtained by using the inner product value of each camera vector {C ₂ , C ₃ } and the light vector g as a coefficient with respect to the pixel values V ₂ and V ₃ . It may be defined by weighted addition. In Equation (1), K is a constant.

<Image generation control means>
The image generation control unit 130 generates an element image group in which pixel values assigned to all pixels of the virtual display are integrated.
When the image generation control unit 130 obtains the assigned pixel value V from the light ray vector g passing through the pixel i, for example, as shown in the equation (1), FIG. 5A, and FIG. The pixel value V is stored in the address corresponding to the pixel i of the virtual display on the image group storage memory 131. That is, in the storage area for the element image group Y on the element image group storage memory 131, the pixel value V is stored at the coordinates corresponding to the corresponding pixel i of the element image G corresponding to the microlens m through which the light vector g passes. . That is, Y (i) = V.

As shown in FIGS. 3 and 4, the microlens m (hereinafter also simply referred to as a lens) of the virtual lens array corresponds to 6 pixels × 6 pixels of the virtual display. Therefore, in this example, each assigned pixel value is stored in accordance with 6 × 6 ray tracing per lens.
In addition, if the number of lenses in the virtual lens array is, for example, a 6 × 6 square array, the process performed for each lens is repeated 36 times. The image generation control unit 130 performs the series of pixel allocation and memory storage processes described above for each pixel of the virtual display, and then outputs the final element image group Y stored in the memory.
Accordingly, when this element array Y is displayed on the two-dimensional display in real space, and this lens array is installed at a position separated by the focal length f of the lenses (microlenses) constituting the lens array, An integral stereoscopic image can be reproduced.

[Operation of Element Image Group Generation Device]
Next, the flow of processing by the element image group generation device 100 will be described with reference to FIG. 6 (see FIGS. 1 and 4 as appropriate). First, in the element image group generation device 100, the ray tracing unit 110 arranges a virtual display, a virtual lens array, and a three-dimensional model in a virtual space (step S1). Then, the image generation control unit 130 sets i = 0 in the pixel i on the virtual display (step S2).

  Then, the ray tracing means 110 searches for the lens (microlens m) nearest to the pixel i in the virtual lens array (step S3), and a straight line L connecting the pixel i and the optical principal point p of the microlens m. Is calculated (step S4), and the light vector g is calculated from the straight line L (step S5). Further, the ray tracing means 110 calculates the intersection point between the straight line L and the three-dimensional model (step S6), and extracts the position of the intersection point S closest to the observer among the plurality of intersection points (step S7).

  Subsequently, the image integration unit 120 selects all cameras in which the angle formed by the camera vector C and the light vector g is equal to or smaller than a predetermined value (step S8). That is, all the cameras whose inner product of the camera vector C and the light vector g is greater than or equal to the threshold value are selected. Then, the image integration unit 120 projects the intersection points S on the selected camera image, and acquires the pixel values thereof (step S9). Next, the image generation control unit 130 stores the pixel value synthesis result at the position of the pixel i in the memory (step S10). Then, the image generation control unit 130 adds 1 to the current value of i (step S11), and determines whether all the pixels have been selected (step S12). If pixels that have not yet been selected remain (step S12: No), the element image group generation device 100 returns to step S3. On the other hand, when all the pixels are selected (step S12: Yes), the image generation control unit 130 outputs an element image group (step S13).

  According to the first embodiment, the element image group generation device 100 selects a camera vector in a shooting direction close to the direction of the light vector g corresponding to the direction of the light beam of the integral stereoscopic image when reproducing the stereoscopic image. Since the intersection S between the ray vector g and the three-dimensional model Q of the subject is projected on the camera image I, the reproduced integral is more effective than the method of selecting the camera closest to the normal line of the three-dimensional model. It is possible to reduce the influence of the error of the three-dimensional model in the stereoscopic image.

(Second Embodiment)
In the first embodiment, the description has been made with reference to FIG. 3 assuming that one model in which the front and side surfaces of the model are shown by a continuous smooth curve is required. However, for example, a subject can be obtained with a depth camera (depth camera). When acquiring the distance image, the subject can be photographed only from one direction. However, if the distance images obtained by photographing the subject from various directions with the depth camera are superimposed, the entire front and sides of the model are handled as one 3D model indicated by a continuous smooth curve. It is possible.

  In the following, a normal color camera is used instead of a depth camera, two cameras are selected from a number of color cameras, and the correspondence between the camera images taken by the camera pair is obtained for each pixel. It is assumed that the depth map of the camera pair has been generated by estimating the depth using the principle.

[Configuration of Element Image Group Generation Device]
Below, the same code | symbol is attached | subjected to the same structure as 1st Embodiment, and description is abbreviate | omitted suitably.
The element image group generation device 100B illustrated in FIG. 7 generates an element image group by arranging a depth map in a virtual space in a computing machine. In the present embodiment, as shown in FIG. 7, it is assumed that a camera image, a camera parameter, a display parameter, a lens array parameter, and a depth map are input from the outside during the processing of the element image group generation device 100B. These input data may be stored in a memory inside the element image group generation device 100B before processing.

  The depth map is a distance image in which the depth to a subject is estimated from some cameras (for example, two stereo cameras) of a multi-viewpoint camera, and depth values for each pixel are arranged two-dimensionally. Since it is necessary to integrate a plurality of distance images in order to generate a three-dimensional model including a side surface or the like, it is preferable to input a plurality of depth maps.

The plurality of depth maps will be described with reference to FIG. 8 (refer to FIG. 7 as appropriate).
The depth map D ₀ is a distance image that is generated using the camera C ₀ and the camera C ₁ and in which distance values from the shooting position of the image I ₀ by the camera C ₀ are arranged.
The depth map D ₁ is a distance image that is generated using the camera C ₁ and the camera C ₀ and in which distance values from the shooting position of the image I ₁ by the camera C ₁ are arranged.
The depth map D ₂ is a distance image that is generated using the camera C ₂ and the camera C ₁ and in which distance values from the shooting position of the image I ₂ by the camera C ₂ are arranged.
The depth map D ₃ is a distance image that is generated by using the camera C ₃ and the camera C ₂ and in which distance values from the shooting position of the image I ₃ by the camera C ₃ are arranged.
The depth map D ₄ is a distance image that is generated using the camera C ₄ and the camera C ₃ and in which distance values from the shooting position of the image I ₄ by the camera C ₄ are arranged.
In the following description, the depth map D is used unless otherwise specified.

As shown in FIG. 7, the element image group generation device 100 </ b> B mainly includes a ray tracing unit 110 </ b> B, an image integration unit 120, an image generation control unit 130, and a depth map control unit 140.
The ray tracing unit 110B outputs the ray vector g calculated by the ray vector calculating unit 112 to the depth map control unit 140, and instead obtains the depth map D selected based on the ray vector g, and obtains the obtained depth map. D is arranged in the vicinity of the virtual display in the virtual space (virtual space information memory 114).

The position information calculation means 113 of the ray tracing means 110B is determined corresponding to the position closest to the observer among the intersections of the straight line L passing through the pixel i of the virtual display and the depth map D arranged in the virtual space. Find the coordinates of the intersection. The obtained coordinates of the intersection point are output to the image integration unit 120. In the example shown in FIG. 8, the position information calculation unit 113 acquires the depth maps D ₂ and D ₃ from the depth map control unit 140, and the intersection S ₀ between the straight line L and the depth map D ₂ , and the straight line L and the depth map. Of the intersection S ₁ with D ₃ , the coordinates of the intersection S _{1 with} the largest Z value are obtained, and the result is output to the image integration means 120.

<Depth map control means>
The camera parameter and the depth map are input to the depth map control means 140.
The depth map control unit 140 includes a camera vector calculation unit 141 and a depth map selection unit 142.

The camera vector calculation unit 141 calculates camera vectors indicating the shooting directions of a plurality of cameras arranged in the virtual space based on the input camera parameters. The calculated camera vectors C ₀ , C ₁ , C ₂ , C ₃ , C ₄ are output to the depth map selection unit 142.

The depth map selection means 142 receives camera vectors C ₀ , C ₁ , C ₂ , C ₃ , and C _{4 and} inputs depth maps D ₀ , D ₁ , D ₂ , D ₃ , and D ₄ .
The depth map selection unit 142 acquires the ray vector g from the ray tracking unit 110B, calculates the inner product of the ray vector g and the camera vectors C ₀ , C ₁ , C ₂ , C ₃ , and C _4. A camera vector that is equal to or greater than a predetermined threshold is determined. Here, the depth map D ₀ to D _4, since in correspondence with the photographing position of the camera image I ₀ ~I _4, respectively correspond to the camera vector C ₀ -C _4. That is, the camera vector determined by the depth map selection unit 142 corresponds to the depth map on a one-to-one basis. In short, the depth map selection unit 142 can determine a depth map in which the inner product of the light vector g and the camera vector is equal to or greater than a predetermined threshold.
In the example shown in FIG. 8, the depth map selection unit 142 calculates the depth maps D ₂ and D ₃ from the depth maps D _{0 to} D ₄ as a result of the arithmetic processing using the ray vector g acquired from the ray tracking unit 110B. Select and output to ray tracing means 110B.

[Operation of Element Image Group Generation Device]
Next, the flow of processing by the element image group generation device 100B will be described with reference to FIG. 9 (see FIGS. 7 and 8 as appropriate).
First, in the element image group generation device 100B, the ray tracing unit 110B arranges a virtual display, a virtual lens array, and a plurality of depth maps D _{0 to} D ₄ in a virtual space (step S1B).
Since each process of the following step S2-step S5 is the same as the flowchart of FIG. 6, description is abbreviate | omitted.
Subsequent to step S5, the depth map control unit 140 uses the calculated camera vector C and the ray vector g acquired from the ray tracking unit 110, and the depth map in which the inner product of these vectors is equal to or greater than a threshold value. D (D ₂ , D ₃ ) is selected (step S21).
Then, the position information calculating unit 113 of the ray tracing unit 110B calculates the intersection of the straight line L and the depth map D (D ₂ , D ₃ ) (step S22).
Since each processing of the following step S7-step S13 is the same as the flowchart of FIG. 6, description is abbreviate | omitted.

  According to the second embodiment, the element image group generation device 100B selects a camera vector in the shooting direction close to the direction of the light vector g corresponding to the direction of the light beam of the integral stereoscopic image when reproducing the stereoscopic image. Since the intersection of the light vector g and the depth map D of the subject is projected on the camera image I, the reproduced integral stereoscopic image is more than the method of selecting the camera closest to the normal line of the three-dimensional model. The effect of errors in the three-dimensional model can be reduced.

The element image group generation device according to the present invention has been described above based on the embodiments, but the present invention is not limited to these. The modifications are listed below.
For example, in each of the above embodiments, the camera selection unit 122 of the image integration unit 120 selects a camera whose angle between the camera vector C and the light vector g is equal to or smaller than a predetermined value. When the angles formed with the vector g are rearranged in ascending order, cameras having a predetermined order or higher may be selected. That is, when the inner product of each camera vector C _n and the light vector g is obtained, and the inner products are rearranged in descending order, the cameras having the inner products of a predetermined order or higher may be selected.

In the second embodiment, the depth map selection unit 142 selects the depth map D such that the inner product of the light vector g and the camera vector C is equal to or greater than a predetermined threshold value, and the light ray tracking unit 110B selects the depth map. Although the coordinates of the intersection determined by using the depth map D selected by the means 142 are obtained, the present invention is not limited to this. For example, the depth map selecting unit 142 selects the depth map D such that when the inner products of the camera vectors C _n and the light vector g are rearranged in descending order, the inner product is _equal to or higher than a predetermined order, and the ray tracing unit is selected. 110B may obtain the coordinates of the intersection determined using the selected depth map D. That is, the ray tracing unit 110B uses the depth map D generated using the cameras of a predetermined order or higher when the angles formed by the camera vector C and the ray vector g are rearranged in descending order. The coordinates may be obtained.

In the second embodiment, for example, when an intersection point is located at a place where an error is likely to occur, such as an end of the depth map D, each intersection point {S ₀ , D of the depth map {D ₂ , D ₃ } and the straight line L If _{one of} S ₁ } having the highest Z value is simply selected, there is a concern of selecting an inappropriate depth map. Accordingly, the following modification may be made so that an appropriate depth map with high probability can be selected.

  In the case of this modification, the ray tracing means 110B obtains, for each depth map D, the intersection closest to the observer among the intersections of the depth map D and the straight line L, and finds the obtained intersections for the selected plurality of camera images I. The similarity of the pixel value at each projection point when projected onto each intersection is calculated for each intersection, and the intersection with the highest calculated similarity is determined as the intersection corresponding to the position closest to the observer.

Specifically, in the example shown in FIG. 8, the ray tracing means 110B selects the intersection S ₀ having the highest Z value in the depth map D ₂ and the intersection S ₁ having the highest Z value in the depth map D ₃ . Select. The image integration unit 120 projects the intersection point S ₀ onto the image I ₂ of the selected camera C ₂ and also onto the image I ₃ of the selected camera C ₃ . The image integration unit 120 uses the pixel values {I ₂ (J ₂ (S ₀ )), I ₃ (J ₃ (S ₀ ))) of the projection points {J ₂ (S ₀ ), J ₃ (S ₀ )}. } Is calculated.
Similarly, the image integration unit 120 projects the intersection S ₁ onto the image I ₂ of the selected camera C ₂ and also onto the image I ₃ of the selected camera C ₃ . The image integration means 120 uses the pixel values {I ₂ (J ₂ (S ₁ )), I ₃ (J ₃ (S ₁ )) of the projection points {J ₂ (S ₁ ), J ₃ (S ₁ )}. } Is calculated.
In this case, ray tracing means 110B from the image integration unit 120 obtains the respective similarity, who similarity in projecting the intersection S ₁ is, higher than the similarity in projecting the intersection S _0, the intersection S ₁ is output to the image integration unit 120 as the determined intersection.
Here, since the intersection point S ₁ has been selected as the one having the highest Z value in the depth map D ₃ , the image integration unit 120 takes an image of the two cameras used when generating the depth map D _3. The pixel value of the pixel corresponding to the projection point J is extracted from the image I ₃ captured by the camera C ₃ at the position, and the extracted value is set as the pixel value V.
Then, the image generation control unit 130 inputs the pixel value V to the coordinates of the corresponding pixel i in the element image group Y. That is, Y (i) = V = I ₃ (J).

In each of the above-described embodiments, the example of tracing light rays one by one as the ray tracing means 110 and 110B has been described. However, the ray tracing method is not limited to this, and is described in, for example, Reference 3 below. A technique may be used.
Reference 3: Y. Iwadate and M. Katayama, “Generating integral image from 3D object by using oblique projection,” Proc. Of International Display Workshops, 3Dp-1, pp. 269-272, 2011.

The method described in Reference 3 uses the parallel projection processing of the computing unit to acquire oblique projection images, thereby simultaneously acquiring rays in the same direction. In this case, it is not necessary to select a camera for each ray, and it is only necessary to select a camera once when acquiring an oblique projection image. That is, the ray tracing means 110 and 110B calculate a ray vector as a ray in the same direction passing through the optical principal point of each lens constituting the virtual lens array by performing oblique projection on the virtual space, and the image integration means 120. Every time oblique projection is performed, a process of selecting a camera is performed. By doing so, it is possible to speed up the processing and reduce the amount of calculation.
For example, if the number of lenses of the virtual lens array is a 6 × 6 square array, for example, when tracing one ray in a certain direction in each element image one by one, the number of lenses (36) is repeated. Necessary processing can be completed once. More specifically, as a process for acquiring a light beam in a certain direction, a process for acquiring the upper left pixel of one element image G as shown in FIG. 5B will be described. Normally, it is necessary to perform the process of acquiring the upper left pixel of the element image G for all element images. However, if the speed-up process for acquiring the oblique projection image, the pixel value of the upper left pixel of all the element images can be acquired by one process.

  By acquiring the oblique projection image, in the second embodiment, the depth map selection unit 142 indicates that the one ray vector g acquired from the ray tracking unit 110B indicates the oblique projection direction of all the lenses of the virtual lens array. Similarly, every time an oblique projection image is acquired, the depth map needs to be selected only once, and does not need to be performed for each ray. The image integration unit 120B may switch and use the depth map selected at that time each time oblique projection is performed.

  When acquiring an oblique projection image, in the first embodiment, for combining a plurality of selected camera images, the selected camera image is mapped to a three-dimensional model, and the oblique projection image is acquired. The processing can be combined after the number of selected cameras is repeated. Computer graphics functions also have such operations (for example, the OpenGL blend function).

When acquiring an oblique projection image, the selection of the intersection point on the plurality of depth maps in the second embodiment is performed by using the Zbuffer in which the depth value is projected, and the intersection point {S ₀ , S ₁ } can be obtained respectively. Here, as in the example shown in FIG. 8, the intersection having the highest Z value in the depth map D ₂ is S ₀ , and the intersection having the highest Z value in the depth map D ₃ is S ₁ . For the selection of the intersection point S based on the similarity determination of the pixel value at the projection point, the image I ₂ of the camera C ₂ is mapped to the depth map {D ₂ , D ₃ }, respectively, and the image I _{3 of the} camera C ₃ is mapped. Is mapped to the depth map {D ₂ , D ₃ }, and the pixel values of the projection points of the similar high depth map are used on the images obtained by obliquely projecting the depth maps.

Here, as the similarity indicating that the similarity is high on each image obtained by obliquely projecting each depth map, the smaller the angle between each camera vector C _n and the light vector g is, the higher the similarity is. Define similar similarity. That is, when comparing the pixel values on each image, the similarity is defined so as to indicate that the similarity is higher as the difference between the pixel values is smaller.
For example, an image Hc ₂ d ₂ obtained by obliquely projecting the depth map obtained by mapping the image I ₂ on the depth map D ₂ , an image Hc ₃ d ₂ obtained by obliquely projecting the depth map obtained by mapping the image I ₃ on the depth map D ₂ , Prepare. Then, the similarity R ₂ can be obtained by calculating the difference between the pixel value of the image Hc ₂ d _{2 at} the target pixel k and the pixel value of the image Hc ₃ d _{2 at} the target pixel k.

Similarly, an image Hc ₂ d ₃ obtained by obliquely projecting the depth map obtained by mapping the image I ₂ onto the depth map D ₃ , and an image Hc ₃ d ₃ obtained by obliquely projecting the depth map obtained by mapping the image I ₃ onto the depth map D ₃ Prepare. By calculating the difference between the pixel value of the image Hc ₂ d _{3 at} the target pixel k and the pixel value of the image Hc ₃ d _{3 at} the target pixel k, the similarity R ₃ can be obtained. If the relationship of R ₂ <R ₃ is established, the intersection point S ₀ on the depth map D ₂ may be output as the determined intersection point.

In the second embodiment, the camera vector calculation unit 141 of the depth map control unit 140 may be replaced by the camera vector calculation unit 121 of the image integration unit 120, or the depth map selection unit 142 may be incorporated in the ray tracing unit 110B. Good.
If the distance images acquired by photographing the subject from various directions with the depth camera described in the second embodiment can be precisely superimposed to generate the three-dimensional model Q shown in FIG. 3, then the first embodiment is performed. You may make it produce | generate an element image group by the method demonstrated by the form.

  In each of the embodiments, the element image group generation apparatuses 100 and 100B have been described. However, the element image group generation program described in a general-purpose or special computer language may be considered so as to enable processing of the configuration of each apparatus. Is possible.

  The element image group generation devices 100 and 100B can present an integral stereoscopic image by using a two-dimensional display and a lens array together. Furthermore, for example, it can be used as a digital broadcast receiving apparatus (integral type stereoscopic television) having a digital broadcast receiving function, a two-dimensional display, and a lens array.

In each of the above embodiments, the case of an integral stereoscopic image that can reproduce both horizontal and vertical parallax has been described. However, a lenticular type that reproduces only horizontal parallax using a kamaboko type one-dimensional microlens. The three-dimensional image can be generated in the same manner.
In addition, by performing the processing described in each of the embodiments on a plurality of frames, a stereoscopic image moving image can be generated.

Furthermore, in the description of each of the above embodiments, the three-dimensional model and the depth map have been described as being generated from a multi-viewpoint video. However, the present invention is not limited to this. For example, the TOF (Time Of Flight) method or the structured illumination method ( A 3D model or a depth map may be acquired using a non-multi-view video such as Structured Light).
For example, when a three-dimensional model or a depth map is acquired by the TOF method, the subject may be captured by a TOF camera and a distance image including distance information to the subject may be acquired.
For example, when a three-dimensional model or a depth map is acquired by the structured illumination method, the subject is captured by a camera used in the structured illumination method, and a monochrome image including luminance information of the subject is acquired. Note that there is no change in photographing a subject with a multi-viewpoint camera (color camera) in order to generate an element image group. Therefore, for example, a TOF camera or the like may be installed in the vicinity of each color camera to acquire a color image of the subject, a distance image, and the like.

DESCRIPTION OF SYMBOLS 100,100B Element image group production | generation apparatus 110,110B Ray tracking means 111 Virtual space setting means 112 Ray vector calculation means 113 Position information calculation means 114 Virtual space information memory 120 Image integration means 121 Camera vector calculation means 122 Camera selection means 123 Pixel value Assigning means 124 Projecting means 125 Compositing means 130 Image generation control means 131 Element image group storage memory 140 Depth map control means 141 Camera vector calculation means 142 Depth map selection means

Claims (8)

An element image group generation apparatus that generates an element image group from a multi-view camera image obtained by photographing a subject with a multi-view camera using a pre-generated three-dimensional model of the subject,
A ray vector which is a direction vector of a straight line representing a light ray emitted from a virtual display pixel arranged in the virtual space and passing through the lenses of the virtual lens array is calculated, and the straight line and the three-dimensional model arranged in the virtual space are calculated. Ray tracing means for obtaining coordinates of the intersection determined corresponding to the position closest to the observer among the intersections;
Based on camera parameters of a multi-viewpoint camera used for photographing the subject, a camera in which an angle formed by a camera vector indicating a photographing direction by a plurality of cameras arranged in the virtual space and the light vector is equal to or smaller than a predetermined value is selected. Image integration means for projecting the coordinates of the intersection point on the camera image input for the selected camera, and assigning a pixel value of the projection point or a pixel value obtained by synthesizing them to the pixel of the virtual display; ,
An element image group generation device, comprising: an image generation control unit that generates an element image group in which pixel values assigned to all pixels of the virtual display are integrated. An element image group generation device that generates an element image group from a multi-view camera image obtained by photographing a plurality of depth maps of a subject using a multi-view camera using a plurality of depth maps generated in advance.
Computes a ray vector, which is a directional vector of a straight line representing a ray emitted from a pixel of the virtual display arranged in the virtual space and passing through the lenses of the virtual lens array, and intersects the straight line and the depth map arranged in the virtual space Ray tracing means for obtaining the coordinates of the intersection determined corresponding to the position closest to the observer,
Based on camera parameters of a multi-viewpoint camera used for photographing the subject, a camera in which an angle formed by a camera vector indicating a photographing direction by a plurality of cameras arranged in the virtual space and the light vector is equal to or smaller than a predetermined value is selected. Image integration means for projecting the coordinates of the intersection point on the camera image input for the selected camera, and assigning a pixel value of the projection point or a pixel value obtained by synthesizing them to the pixel of the virtual display; ,
An element image group generation device, comprising: an image generation control unit that generates an element image group in which pixel values assigned to all pixels of the virtual display are integrated.   3. The element image group according to claim 1, wherein the image integration unit selects a camera having a predetermined order or higher when the angles formed by the camera vector and the light vector are rearranged in ascending order. Generator.   When the ray tracing means rearranges the angle formed by the camera vector and the ray vector in ascending order, the coordinates of the determined intersection point using a depth map generated using a camera of a predetermined order or higher The element image group generation device according to claim 2, wherein:   The ray tracing means obtains, for each depth map, an intersection closest to the observer among intersections of the depth map and the straight line, and each projection point when the obtained intersection is projected onto the selected plurality of cameras. 3. The element image group according to claim 2, wherein a similarity of pixel values is calculated for each intersection, and an intersection having the highest similarity is determined as an intersection corresponding to a position closest to the observer. Generator. The ray tracing means performs oblique projection on a virtual space and calculates the ray vector as a ray in the same direction passing through the optical principal point of each lens constituting the virtual lens array,
The element image group generation device according to claim 1, wherein the image integration unit performs a process of selecting the camera every time the oblique projection is performed. The ray tracing means performs oblique projection on a virtual space and calculates the ray vector as a ray in the same direction passing through the optical principal point of each lens constituting the virtual lens array,
The element image group generation device according to claim 2, wherein the image integration unit switches the depth map each time the oblique projection is performed.   An element image group generation program for causing a computer to function as the element image group generation device according to any one of claims 1 to 7.

Images