JP6502701B2 - Element image group generating device, program therefor, and digital broadcast receiving device - Google Patents

Description

  The present invention relates to an element image group generation apparatus for 3D television using an integral system that can be viewed from any viewpoint, a program thereof, and a digital broadcast receiving apparatus.

The stereoscopic video system can be roughly classified into, for example, a two-eye system, a multi-eye system, and a spatial image reproduction system. Here, the space image reproduction method is a method for reproducing an object as an optical space image, and an integral method and a holography method are representative methods.
In the two-lens method, a stereoscopic image is perceived by presenting images corresponding to the left and right eyes. The twin-lens method has a drawback that it can not reproduce motion parallax because it uses only a pair of images corresponding to the left and right eyes.

The multi-view method is a method of presenting images of multiple viewpoints arranged side by side to left and right eyes using a lenticular lens or the like, and can reproduce motion parallax associated with horizontal viewpoint movement. In this multi-view system, the number of viewpoints is often about eight, and the number of parallax images is larger than that of the two-view system, but the principle is the same as that of the two-view system.
The holographic method is a method for recording and reproducing the wavefront of light from an object, and although it has the advantage of being able to reproduce an ideal spatial image, the display requires a larger number of pixels than the integral method. There is a drawback.

  The integral method is a method of recording and reproducing an optical spatial image of an object using an imaging and display device configured of a high definition image system and a lens group (lens array). Since the aerial image is reproduced, motion parallax in the horizontal and vertical directions can be reproduced. Moreover, compared to the number of viewpoints of the multiview system being about eight, the number of viewpoints of the integral system is, for example, 20 × 20, which is an extraordinary number, and the same conditions as viewing the real thing are satisfied. Adjustment points coincide.

  In addition, it is possible to classify whether the stereoscopic image display method is a display method requiring special glasses such as 3D glasses or an autostereoscopic display method. Along with this, video contents reproduced by the stereoscopic video display method can be similarly classified. For example, a twin-lens system that uses 3D glasses to view 3D movies is a stereoscopic display system that has been put to practical use, and a 3D movie that has been put to practical use has video contents of display methods that require special glasses It is.

  On the other hand, the integral method is an autostereoscopic display method. And an integral system is expected as one of the imaging systems for the future 3D television which can be visually recognized from arbitrary viewpoints, and a lens group on a plane or a spherical surface is used for the lens array to be used. Are arranged. FIG. 15 is an explanatory view schematically showing a configuration from shooting to display of the stereoscopic television system adopting the integral system. In recent years, a method of generating an element image group from multi-view images captured by a plurality of cameras has also been proposed.

  In the stereoscopic television system shown in FIG. 15, the image of the subject 201 is formed in the vicinity of the lens array 203 via the convex lens 202. Further, this image formation is photographed by the television camera 204, and an output signal after being converted into an output of an image signal is displayed on the liquid crystal panel 205. The lens array 206 is also provided in the vicinity of the liquid crystal panel 205, so that the operator can visually recognize a stereoscopic image without wearing special glasses by the image through the lens array 206. In this example, an image (image of the subject 201) seen in front of the lens array 206 is integral video content.

  Reference numeral 207 in FIG. 15 shows a magnified image of an image obtained on the lens array 206, for example. As can be seen from the image enlarged view 207, one of images obtained by photographing the subject 201 from various angles is projected as an element image on each of the element lenses included in the lens array 206. That is, the output signal of the television camera 204 is obtained by integrating element image signals obtained from the lens group of the lens array 203.

Here, the resolution of the three-dimensional image at the time of shooting is the number of television cameras 204 used for shooting, or the number of imaging elements of the television camera 204 and lenses constituting the lens group of the lens array 203 (hereinafter referred to as element lenses It is known to relate to the number of
In the future, for example, the number of element lenses constituting the lens group of the lens array 203 increases, and the number of pixels (hereinafter referred to as element image) allocated to one element lens (number of pixels) The resolution of the three-dimensional image at the time of photography will also increase. Similarly, even when displaying a solid image by the integral method, in order to display a natural three-dimensional image, the lens group of the lens array 206 requires more element lenses, and in the element image, Is considered to require more pixels.

Also, among the autostereoscopic display methods that have been proposed or prototyped conventionally, as the autostereoscopic method other than the integral method, for example, a parallax barrier method based on binocular stereoscopic vision or a multiview method A lenticular method is known which is based on three-dimensional vision.
An image projected on a screen, for example, to display a stereoscopic image through such a lenticular lens, or an image displayed on a liquid crystal panel, for example, to display a stereoscopic image through a parallax barrier It is generated based on the multi-viewpoint image of the subject that is the source of. Therefore, image processing for converting an original multi-viewpoint image into an image suitable for a display device of each system is required. Similarly, even in a two-eye system requiring special glasses, a process of converting the original multi-viewpoint image into an image adapted to the Active system or the Passive system is required. Such image processing is greatly influenced by the specifications of the display device.

  When a lenticular lens is used, for example, images of eight viewpoints arranged in the lateral (horizontal direction) are the original multi-viewpoint images before being converted into an image suitable for a lenticular display device. Although this multi-viewpoint image is the source of video content, here, the multi-viewpoint image itself that is the source is also referred to as video content. The video content of naked-eye three-dimensional methods (lenticular method, parallax barrier method) other than the integral method, like the integral-type video content, currently has a smaller number of contents than the two-eye method video content. is there.

  On the other hand, conventionally, there has been known a video processing apparatus capable of selecting a two-view system or a multiview system having three or more parallaxes as a stereoscopic video display system according to the 3D viewing mode (see Patent Document 1). The video processing apparatus described in Patent Document 1 includes a liquid crystal panel and a lenticular lens, and when an input signal of a twin-lens type video content is input, it is reproduced as a twin-lens type video as it is, or After converting an input signal, it reproduces as an image suitable for a multi-view system with three or more parallaxes. By using this technology, it is possible (apparently) to increase the number of lenticular video content.

Patent No. 5129376

  Future television broadcasts are expected to require a large number of video content that can be displayed by an integral display that receives digital broadcasts. Therefore, it is considered to convert image content such as twin-lens type with a large number of contents into image content that can be displayed on an integral type display and to increase the number of image contents that can be presented in integral type. Be However, conventionally, no technology has been known for converting an input signal such as a two-lens system into an image suitable for an integral display.

  The image processing apparatus described in Patent Document 1 relates to a lenticular method, and the principle is largely different from the integral method, and therefore, it can not be converted into an image suitable for an integral method display. The reason is that the integral type display is premised on the existence of a large number of element images (element image group) as indicated by reference numeral 207 in FIG. 15, while the twin-view type and multi-view type video contents This is because the content corresponds to only the horizontal parallax.

  The present invention has been made in view of the above problems, and it is possible to efficiently increase the number of video contents that can be displayed on an integral type display, an apparatus for generating an image group, a program thereof, and a digital It is an object of the present invention to provide a broadcast receiving apparatus.

  In order to solve the above problem, an element image group generation device according to the present invention is configured to display element images of multiple integral element images displayed on a display panel to present a stereoscopic image through a lens array. An element image group generation apparatus for generating, comprising viewpoint synthesis processing means, multi-view video correction means, video switching means, element image generation means, and multi-view video recording and transfer means.

According to this configuration, the element image group generation device receives the video composed of the respective viewpoint videos of the plurality of viewpoints corresponding to only the horizontal parallax between the viewpoints by the viewpoint synthesis processing means. Based on lens array information including information on lens pitch of lens array, the viewpoints are synthesized from a plurality of viewpoint videos having the horizontal parallax, and the viewpoint video is formed by increasing the number of viewpoints compared to the input video. Generate multi-view video.
Then, the element image group generation apparatus corrects the multi-viewpoint image generated by the viewpoint synthesis processing means into an image of a projection system suitable for the integral-type element image generation method by the multi-view image correction means.
Then, the element image group generation apparatus selects a predetermined multi-view image by the image switching means and sets the multi-view image, which is a multi-view image per unit time, to the horizontal of the multi-view image arrangement area defined on the memory. A process of generating a multi-viewpoint image group including a plurality of multi-viewpoint images, which are allocated in the direction and the vertical direction, is repeatedly performed in time division.
Then, the element image group generation device generates the element image group by repeating the process of generating an element image from the multi-viewpoint image group by the element image generation method of the integral method by the element image generation means.
Then, the element image group generation apparatus records the multi-viewpoint image generated by the correction of the multi-view image correcting unit by the multi-view image recording and transferring unit in the recording buffer and transfers it from the recording buffer to the image switching unit. .
Then, the element image group generation device is configured to, based on the multi-view video transferred from the multi-view video recording and transfer means by the video switching means, with respect to the horizontal direction of the multi-view image arrangement area, To assign the viewpoint video of the horizontal component generated by the process, and to repeatedly generate the multi-viewpoint image group by repeatedly assigning the viewpoint video at the same horizontal position in the vertical direction of the multi-viewpoint image arrangement area. It is characterized by

  The present invention can also be realized by an element image group generation program for causing a computer to function as the element image group generation device.

  Further, in order to solve the above problems, a digital broadcast receiving apparatus according to the present invention includes the element image group generating apparatus, the display panel, the lens array, a broadcast receiving unit for receiving a broadcast wave, and a communication network. And D. communication means for transmitting and receiving information to and from the outside via the I. A digital broadcast receiving apparatus comprising: a display switching means.

  According to this configuration, when the digital broadcast receiving apparatus receives the multiview video with only horizontal parallax from the broadcast wave by the broadcast receiving section by the display switching section, the digital broadcast receiving apparatus changes the multiview video with only the horizontal parallax received by the broadcast. Based on the display of the integral stereoscopic image having no vertical parallax generated on the basis and the multi-view image other than the multi-view image of only the horizontal parallax received by the broadcast wave by the communication means via the communication network and the network It switches between displaying the integral stereoscopic image having both horizontal parallax and vertical parallax generated by using the received multi-viewpoint image in combination.

  In addition, when generating an integral stereoscopic image having both horizontal parallax and vertical parallax, multi-viewpoint video of only horizontal parallax received in the broadcast wave is recorded in advance in a recording device such as HDD, and only the horizontal parallax recorded. The multi-view video of the present invention and the multi-view video received via the communication network at different times from the reception of the broadcast wave may be used in combination.

  According to the present invention, it is possible to generate an integral-type element image group from an image that supports only horizontal parallax. Therefore, according to the present invention, it is possible to efficiently increase the number of video contents that can be displayed on an integral display required for future television broadcasting.

It is a block diagram which shows typically the structure of the element image group production | generation apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing of operation | movement of the element image group production | generation apparatus of FIG. 1, Comprising: (a) is an example of an input image, (b) shows an example of a storage structure of a recording buffer, (c) shows an example of allocation of a viewpoint video. There is. It is explanatory drawing of operation | movement of the image | video switching means of the element image group production | generation apparatus of FIG. 1, Comprising: It is a schematic diagram of an example of a multi-viewpoint image group. It is explanatory drawing of operation | movement of the element image generation means of the element image group production | generation apparatus of FIG. 1, Comprising: It is a schematic diagram of an example of an element image group. It is explanatory drawing of operation | movement of the element image group production | generation apparatus of FIG. 1, Comprising: (a)-(c) has shown an example of the transition of the viewpoint image allocation of a perpendicular | vertical component. It is a conceptual diagram which shows an example of how to see the solid image in the boundary of a viewing zone. It is a top view which shows typically a mode when the boundary of a viewing area is located between right eye and left eye. It is a graph which shows typically a mode that the viewpoint imaging | video in multi-viewpoint imaging | video switches discontinuously at the boundary of a viewing zone. It is a block diagram which shows typically the structure of the element image group production | generation apparatus which concerns on 2nd Embodiment of this invention. It is explanatory drawing of the method of producing | generating an element image in virtual space by the element image group production | generation apparatus of FIG. 9, Comprising: (a) is oblique projection for every light ray which passes an element lens, (b) passes a plurality of element lenses. Show parallel light. FIG. 10 is a graph schematically showing an example of selecting multi-viewpoint images and replacing the images so as to give continuity at boundaries of viewing regions by the element image group generation device of FIG. 9; FIG. FIG. 10 is a graph schematically showing another example of selecting multi-viewpoint images and replacing the images so as to give continuity at boundaries of viewing regions by the element image group generation device of FIG. 9; FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows typically the broadcast communication cooperation system containing the digital broadcast receiver which concerns on embodiment of this invention. It is a block diagram which shows typically the digital broadcast receiver containing the element image group production | generation apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows typically the structure of the conventional integral-type three-dimensional television camera.

  Hereinafter, an embodiment (hereinafter referred to as “embodiment”) for carrying out an element image group generation apparatus of the present invention will be described in detail with reference to the drawings.

[1-1. Configuration of Element Image Group Generation Device]
The element image group generation apparatus 10 shown in FIG. 1 generates an element image group which is a plurality of element images necessary for the integral method to present a stereoscopic image. The element image group generation apparatus 10 is constituted by, for example, an element image group generation program or a computer incorporated in a general computer, and presents a stereoscopic image by using a monitor (display panel) and a lens array in combination. Can. Alternatively, although the details will be described later, as shown in FIG. 14, for example, it can be used as a digital broadcast receiving apparatus 3 (integral stereo television) provided with a monitor 70 and a lens array 80.

  In the following, the element image group generation device 10 has the same components as a general computer, that is, for example, a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD) Shall be provided. Here, the element image group generation apparatus 10 performs, as shown in FIG. 1, the input interface 11, the lens array information storage unit 12, the viewpoint synthesis processing unit 13, and the multi-view image correction to perform its functions. A unit 14, a multiview video record transfer unit 15, a video switching unit 16, an element image generation unit 17, and an output interface 18 are provided.

  The input interface 11 is a predetermined interface for inputting an input video V from the outside. The input video V is input by, for example, a disk drive, wired communication using a predetermined cable or the like, wireless communication, broadcast waves, or the like.

The input video V may be a moving image or a still image. The input video V is, for example, a multi-view video which is a set of viewpoint videos such as a twin-view method, a multi-view method, or an integral method.
Here, the viewpoint video is a video of a camera or a video of a virtual camera when a subject is viewed from a certain viewpoint. The viewpoint video is one of two parallax videos corresponding to binoculars in the two-view system. In the multiview method, the viewpoint video having horizontal parallax is three or more, for example, one of eight parallax videos. In the integral method, a plurality of viewpoint videos are present in each of the horizontal and vertical directions. For example, a camera array or a virtual camera having a three-stage configuration in which the lower row is 2 viewpoints: the middle row is 3 viewpoints and the upper row is 2 viewpoints is assumed.

  Here, it is assumed that the multiple viewpoints are two or more viewpoints. In the case of multi-view video, in addition to the stereoscopic display method assuming that there are three or more viewpoints in the multi-view video, even in the twin-lens method, the number of viewpoints is increased to perform the view synthesis processing. When it is called multi-view video. Furthermore, a group of a plurality of multi-view videos assumed in space in the stereoscopic display system or a system in which the number of viewpoints is increased is also referred to as a multi-view video group (of the same frame).

When the input video V is an integral-type multi-view video, the input video V has both horizontal parallax and vertical parallax. When the input video V is a twin-view multi-view video (two parallax videos) or a multi-view multi-view video, the input video V has only horizontal parallax.
In addition, when referring to a stored image, an image in a frame unit, or the like with respect to a viewpoint video or a multi-view video, it may be referred to as a viewpoint image or a multi-view image.

  The lens array information storage unit 12 stores lens array information including information on the lens pitch of the lens array 80, and can be configured by, for example, a general storage medium such as a semiconductor memory. The lens array information may include, for example, the position of the center of each lens, the size of the lens, the number of lenses, the lens arrangement pattern, the distance from the viewpoint of the virtual camera to the lens array surface, and the like.

  The viewpoint composition processing means 13 sets lens array information including information on the lens pitch of the lens array when an image composed of viewpoint images of a plurality of viewpoints corresponding to only horizontal parallax between viewpoints is input. Based on this, viewpoints are synthesized from a plurality of viewpoint videos having the horizontal parallax, and a multi-viewpoint video composed of viewpoint videos in which the number of viewpoints is increased compared to the input video V is generated. The total number of viewpoints after the increase can be determined in advance based on the information of the lens array (integral display information).

  In the present embodiment, the viewpoint synthesis processing unit 13 determines the necessity of viewpoint synthesis from the input video V. As this determination method, for example, when the number of viewpoint videos included in the same frame of the input video V is less than the set number of multiview videos forming the multiview video group generated in the later stage, it is necessary to combine viewpoints. It may be determined that there is. As an example, the setting number of multi-view videos is set in advance to the condition of the number of multi-view videos of integral type (for example, 20 horizontal × 20 vertical) and the same frame of the input video V If the number of included viewpoint videos satisfies the condition, it is determined that the viewpoint combining process is unnecessary for the input video V, and if the condition is not satisfied, it can be determined that the viewpoint combining process is necessary. The above determination is not essential, and information on whether or not viewpoint combination is necessary may be given to the viewpoint combination processing means 13 in advance.

Then, when the view synthesis process is unnecessary, that is, when the input video V is an integral multiview video, the view synthesis processing unit 13 passes the process to the multiview image correction unit 14 without performing the view synthesis.
On the other hand, when the view synthesis processing is necessary, that is, when the input video V is a twin-view or multi-view multi-view video, the view synthesis processing means 13 is the source of the twin-view or multi-view video content. In order to convert a multi-viewpoint image into a multi-viewpoint image that is the source of integral-type video content, view synthesis is performed by referring to information of the lens array based on a plurality of images with horizontal parallax. Increase the score. Then, the generated multi-view video is output to the multi-view video correction unit 14.

Here, viewpoint synthesis means processing of viewpoint interpolation in which one interpolated video is created between each viewpoint video of the input video V, and a plurality of interpolated videos are created between each viewpoint video. It is. The number of viewpoints may be determined based on the information of the lens array. For example, a method of using a depth map is known as processing of viewpoint interpolation, and this can be used. In addition, about the method of utilizing a depth map, since it describes, for example in the following reference, description is abbreviate | omitted here.
(Reference) ISO / IEC JTC1 / SC29 / WG11 MPEG2013 / M31520

  The multi-view video correction means 14 corrects the multi-view video generated by the view synthesis processing means 13 into a video of a projection system suitable for the integral type elementary image generation method. In the integral type element image generation method, for example, an element image is generated based on an image acquired by oblique projection. Therefore, in the present embodiment, as an example, correction processing is performed to convert each viewpoint video of a multi-view video in which the number of viewpoints has been increased by viewpoint interpolation processing into an oblique projection image.

The multi-viewpoint image correction unit 14 includes an oblique projection processing unit 141 and a first determination unit 142.
The oblique projection processing means 141 performs processing of converting a perspective projection into an image of parallel projection. This is because the content in the two-view method and the multi-view method is usually an orthographic projection because it is an image acquired from a camera.
The oblique projection processing means 141 converts the perspective projection image into a parallel projection image based on lens array information such as the lens pitch, and converts the multi-viewpoint image generated by the viewpoint synthesis processing means 13 into a converted video signal. Are sent to the multi-viewpoint video recording and transferring means 15 and the video switching means 16.

  The first determination means 142 compares the focused multi-viewpoint video with the image data of a frame temporally adjacent to the multi-viewpoint video or the image data spatially adjacent to the multi-viewpoint video with reference to the recording buffer 151. Then, it is determined whether or not the multi-viewpoint video, which is a set of viewpoint videos, input to the element image group generation device 10 for each frame is missing the viewpoint video constituting the video. The first determination means 142 determines that there is a loss of the viewpoint video when there is no data of the multi-view video to be focused as a result of the comparison. The first determination means 142 notifies the multi-view video record transfer means 15 of information on the presence or absence of the dropout. The reason why such a determination is made is that there is a possibility that part of the information in the input video V input to the element image group generation device 10 may be lost in transmission before input to the element image group generation device 10 or the like. It is because there is.

  The multi-viewpoint image correction means 14 performs the processing of the first determination means 142 without performing the processing by the oblique projection processing means 141 when the multi-viewpoint image of the integral type is input, It is sent to the viewpoint video recording and transferring means 15 and the video switching means 16.

The multi-view video record transfer means 15 records the multi-view video generated by the correction of the multi-view video correction means 14 in the recording buffer 151 and transfers the multi-view video from the recording buffer 151 to the video switching means 16 .
The multi-viewpoint video recording and transfer unit 15 performs multi-viewpoint correction processing by the multi-viewpoint image correction unit 14 on the multi-viewpoint video that is the source of the twin-view type or multi-view type video content in the input video V. The video is recorded in the recording buffer 151.
In this embodiment, the multi-viewpoint video recording and transferring unit 15 includes the recording buffer 151, the image output unit 152, and the correcting unit 153.

The recording buffer 151 is for recording multi-view video, and can be configured, for example, by a general storage medium such as a semiconductor memory. Incidentally, the recording of the multi-viewpoint video to the recording buffer 151 is temporary.
The image output unit 152 reads out the multi-view video (multi-view image) from the recording buffer 151 and outputs it to the video switching unit 16.
The relationship between the recording buffer 151 and the image output unit 152 is like a so-called skip back recorder, and temporarily stores multi-view images input to the multi-view image recording / transfer unit 15 in time series to the recording buffer 151. , And delayed for a predetermined time (for example, several seconds) to output a multi-view video (multi-view image) to the video switching means 16.

The correction means (first correction means) 153 is to use the information stored in the recording buffer 151 for correction when there is a defect in the viewpoint video. The corrected information is output by the image output unit 152.
In the present embodiment, when it is determined by the first determination unit 142 that there is a loss of the viewpoint video, the correction unit 153 is adjacent to the image data of the frame temporally adjacent to the deletion portion or spatially adjacent. The image data is read from the recording buffer 151 and used to correct the image of the missing part. For example, the correction unit 153 may use data (null) of the memory area in which the viewpoint image missing in the predetermined frame is arranged on the recording buffer 151, the viewpoint image of the previous frame adjacent to the frame, and the subsequent image. It replaces with the data of the viewpoint image generated by interpolation using the viewpoint image of the frame.

The video switching means 16 receives the video (predetermined multi-view video) sent from the multi-view video correction means 14 or the multi-view video recording and transfer means 15 and outputs the video to the element image generation means 17.
The image switching unit 16 outputs the image sent from the multi-viewpoint image correction unit 14 to the element image generation unit 17 as it is.
The image switching means 16 selects the multi-view image sent from the multi-view image recording and transfer means 15 and sets the multi-view image which is the multi-view image per unit time to the horizontal of the multi-view image arrangement area defined on the memory. A process of generating a multi-viewpoint image group including a plurality of multi-viewpoint images, which are allocated in the direction and the vertical direction, is repeatedly performed in time division.

  In the present embodiment, the video switching unit 16 is generated by the viewpoint synthesis processing unit 13 in the horizontal direction of the multi-viewpoint image arrangement area based on the multi-viewpoint video transferred from the multi-viewpoint video recording and transfer unit 15. A process of generating a multi-viewpoint image group is performed by assigning the viewpoint video of the horizontal component and repeatedly assigning the viewpoint video at the same horizontal position in the vertical direction of the multi-viewpoint image arrangement region. A specific example will be described later.

  The element image generation unit 17 generates an element image group by repeating a process of generating an element image from the multi-viewpoint image group by the integral type element image generation method. As the integral-type element image generation method, for example, a method of generating an integral three-dimensional image by oblique projection can be used. Since this method is described in detail in JP 2012-84105 A, the description is omitted here.

  The output interface 18 outputs the element image group generated by the element image generation unit 17 to, for example, a display panel (monitor 70) such as a liquid crystal display, or outputs it to an accumulation unit such as a hard disk recorder. Note that an integral type lens array 80 is installed on the display panel (monitor 70), and when the display panel displays an element image group, a stereoscopic image is reproduced.

[1-2. Overall Flow of Processing by Element Image Group Generator]
The overall flow of processing by the element image group generation apparatus 10 according to the first embodiment will be described with reference to FIG.

The viewpoint synthesis processing unit 13 of the element image group generation device 10 receives information such as the lens pitch from the lens array information storage unit 12 and determines the necessity of viewpoint synthesis from the input video V.
If it is determined that the view synthesis is necessary, the view synthesis processing unit 13 performs the synthesis process. For example, after creating a view image, the view synthesis processing unit 13 outputs the image to the multiview video correction unit 14 and does not need the view synthesis. An image is output to the multi-viewpoint image correction means 14 without performing the synthesis process. While the input video V is being input, the viewpoint synthesis processing means 13 sequentially repeats the above processing.

  When the multi-viewpoint image correction unit 14 receives the image combined with the viewpoint by the viewpoint combination processing unit 13, the multi-viewpoint image correction unit 14 performs perspective projection on the image combined with the viewpoint based on the information such as the lens pitch from the lens array information storage unit 12. Parallel projection is converted, and the converted video signal is sent to the multi-view video record transfer means 15. In addition, when the multi-viewpoint video correction unit 14 receives a video that does not require viewpoint combination, the multi-viewpoint video correction unit 14 sends it to the multi-viewpoint video recording and transfer unit 15 and the video switching unit 16. Further, when the multi-viewpoint video correction unit 14 receives the video from the viewpoint combination processing unit 13, the multi-viewpoint video correction unit 14 determines whether there is a loss of viewpoint video and notifies the multi-viewpoint video recording and transfer unit 15 of the determination result. While the input video V is being input, the multi-viewpoint video correction means 14 sequentially repeats the above processing.

  The multi-viewpoint video recording and transferring unit 15 records the video sent from the multi-viewpoint video correcting unit 14 in the recording buffer 151 and sends the video to the video switching unit 16. Also, when the multi-viewpoint video correction unit 14 notifies the multi-viewpoint video recording and transfer unit 15 that the viewpoint video is missing, the multi-viewpoint video recording and transfer unit 15 corrects the viewpoint video using the data recorded in the recording buffer 151. Send to While the input video V is being input, the multi-viewpoint video recording and transferring unit 15 sequentially repeats the above process.

  When the video switching means 16 receives the multi-view video from the multi-view video recording and transferring means 15, the video switching means 16 arranges it as a multi-view image in time division, and repeatedly outputs the multi-view image to the element image generating means 17. While the input video V is being input, the video switching means 16 sequentially repeats the above process. Further, the video switching means 16 switches between the multi-view video received from the multi-view video correction means 14 and the multi-view video received from the multi-view video recording and transfer means 15 and outputs the same. This video switching may be switched to a predetermined timing before operation or the like by an instruction operation by the operator of the element image group generation apparatus 10.

  The element image generation unit 17 generates an element image group by repeating the process of generating an element image from the multi-viewpoint image received from the image switching unit 16 according to the integral type element image generation method. While the multi-viewpoint image is input from the video switching unit 16, the element image generation unit 17 sequentially repeats the above process.

[1-3. Processing flow when the input video is an integral-type multi-view video]
A flow of processing by the element image group generation apparatus 10 in this case will be described with reference to FIGS. 2 to 4 (refer to FIG. 1 as needed). Here, as an example, a camera array in which nine cameras are arranged in three lines vertically and three lines horizontally is prepared, and a cylinder and a cube arranged in front of it are photographed as a subject Assume an image of 3 × 3 viewpoints. An example of the input video at this time is shown in FIG. Here, when the subject is viewed from the side of the camera array, the upper left (coordinates = (1, 1); the same applies hereinafter), the front (1, 2), the upper right (1, 3), the middle left (2, 1) ), View images taken from the front (2, 2), right middle (2, 3), lower left (3, 1), lower front (3, 2), lower right (3, 3) It is referred to as V1 to V9. These input videos V1 to V9 are videos of the same frame.

  In the case of the integral-type multi-view video, as shown in FIG. 2A, the multi-view video has both horizontal parallax and vertical parallax. Therefore, the element image group generation apparatus 10 does not perform processing of viewpoint synthesis and oblique projection, and the multi-viewpoint video recording and transfer unit 15 inputs an input video (multi-viewpoint video) as schematically shown in FIG. V1 to V9 are recorded in the recording buffer 151, and the video is sent to the video switching means 16. Then, the image switching means 16 is formed in the memory corresponding to the camera arrangement of the camera array used for photographing, in the horizontal direction and the vertical direction of the multi-viewpoint image arrangement area 61 schematically shown in FIG. Input video (multi-view video) V1 to V9 are assigned to the area of 3 rows and 3 columns while rearranging.

  FIG. 3 is a schematic view of a multi-viewpoint image group 62 generated by the video switching unit 16 allocating input videos (multi-viewpoint videos) V1 to V9 on the memory at this time. The element image generation unit 17 generates an element image group from the multi-viewpoint image group 62 acquired from the image switching unit 16. A normalized element image to be described later may be generated on the way.

  Here, a method of generating an element image from a multi-viewpoint image group will be described with reference to FIGS. 3 and 4. FIG. 4 is a schematic view of an example of the completed element image group 63 corresponding to a certain frame. The six squares surrounded by thick solid lines displayed in FIG. 4 are an example of element images constituting the element image group 63. Each elementary image is composed of an elementary image of 3 pixels × 3 pixels. The area | region divided by the broken line in each element image of FIG. 4 has shown the pixel which comprises the multi-viewpoint image group shown in FIG. Also, the numbers entered in place of the picture of the subject in each element image of FIG. 4 indicate the positions of the pixels constituting the multi-viewpoint image group shown in FIG. 3.

  In the multi-viewpoint image group 62 of FIG. 3, as shown by enlarging a rectangular area of a part of the multi-viewpoint image arranged at the upper left (1, 1), pixels of 2 pixels × 3 pixels in the rectangular area. Is arranged. The numbers entered in place of the picture of the subject in the enlarged rectangular area indicate the positions of the pixels in the rectangular area. The same applies to the other multi-viewpoint images in the multi-viewpoint image group 62, so detailed description will be omitted.

  The element image generation unit 17 generates one element image by collecting pixels at the same coordinate position from each multi-viewpoint image forming the multi-viewpoint image group 62. Specifically, when the pixels arranged at the upper left of the rectangular area of each multi-viewpoint image of FIG. 3 are accumulated while maintaining the entire arrangement of multi-viewpoint image group 62, element image group 63 of FIG. An element image of the upper left corner can be generated. Other elemental images can be generated as well.

[1-4. Flow of processing when input video is multi-view type multi-view video]
A flow of processing by the element image group generation apparatus 10 in this case will be described with reference to FIG. 5 (refer to FIGS. 1 to 4 as needed). Here, as a premise, it is assumed that, for example, a video of 6 × 6 viewpoints is used in the case of the integral method. That is, as schematically shown in FIG. 5A, the image switching unit 16 of the element image group generation apparatus 10 inputs the input image to the horizontal and vertical six rows and six columns of the multi-viewpoint image arrangement region 64. It will be allocated while rearranging (multi-view video). The number of viewpoints may be determined based on the information of the lens array.

  Under this premise, three cameras are prepared in the horizontal direction for multi-view shooting, and images of three viewpoints obtained by shooting a cylinder and a cube placed in front of it as a subject are assumed. Here, these multi-view videos are referred to as the left video (1, 1), the center video (1, 3), and the right video (1, 6). In this case, the viewpoint synthesis processing unit 13 of the element image group generation device 10 increases the number of viewpoints as follows, for example. An image (1, 2) is generated by interpolation at a midpoint between the left image (1, 1) and the center image (1, 3). Further, video (1, 4) and video (1, 5) are generated at two places between the central video (1, 3) and the right video (1, 6) by, for example, linear interpolation. The multi-viewpoint image group of a total of six viewpoints generated by the above-described view synthesis is converted by the multi-viewpoint image correction means 14 from an image of perspective projection to an image of parallel projection. Then, the multi-viewpoint video recording and transfer unit 15 temporarily records the multi-viewpoint video subjected to the correction processing by the multi-viewpoint image correction unit 14 in the recording buffer 151.

  Then, the image switching means 16 is a multi-viewpoint image arrangement area schematically shown in FIG. 5A formed on the memory in correspondence with the number of viewpoints (here, 6 × 6) determined by the information of the lens array. The multi-view video is reordered and assigned to 64 horizontal and vertical regions of 6 rows and 6 columns. At this time, when generating the element image, the element image group generation device 10 has already generated the viewpoint image of the horizontal component by the viewpoint synthesis, so the image switching means 16 is shown in FIG. Can be easily assigned. However, in the case of a multiview type viewpoint video, the viewpoint video is originally a video with only horizontal parallax, and the viewpoint video of the vertical component has only one point for each horizontal component, so accurate viewpoint interpolation is also difficult. Assuming that the multi-viewpoint image group 65 of FIG. 5B is taken as an example, the element image generation unit 17 can not generate an element image group by the method described with reference to FIGS. 3 and 4.

  However, in the element image group generation device 10, for the viewpoint video with only horizontal parallax, the video switching unit 16 assigns the same video stored in the recording buffer 151 as the viewpoint video of the vertical component. Thus, a multi-viewpoint image group 65 schematically shown in FIG. 5C is generated. Thereby, the element image generation unit 17 can generate an element image group by the method described with reference to FIGS. 3 and 4.

  As described above, the element image group generation apparatus 10 according to the present embodiment performs viewpoint synthesis based on viewpoint images with horizontal parallax such as content in the two-lens method or multi-lens method, and the image is integrated After correction processing is performed on an image suitable for an element image generation method based on the oblique projection method, the image is temporarily stored in the recording buffer 151, and the stored image is allocated to generate an element image. Here, with regard to assignment of images of vertical components, viewpoint images at the same horizontal position are repeatedly used.

  Therefore, according to the element image group generation device 10, it is possible to display the video content corresponding only to the horizontal parallax image on the integral stereoscopic display. In short, it is possible to display a two-view or multi-view video content on an integral display. As a result, it is possible to efficiently increase the number of video contents that can be displayed on the integral display.

  In addition, the element image group generation device 10 includes the recording buffer 151, and by using this as a skip back recorder, transmission of the input video V before being input to the element image group generation device 10, etc. Even if a part of information (viewpoint image) of the multi-viewpoint image is missing, the viewpoint image can be generated by interpolation.

Second Embodiment
The element image group generation apparatus according to the second embodiment has a function of displaying a more natural three-dimensional image in addition to the configuration of the element image group generation apparatus according to the first embodiment. Before describing the apparatus configuration, the background of the additional function and the problem to be solved will be described in detail.

  As a display method for stereoscopic television in the future, in addition to the viewing mode at an individual viewer home, for example, a digital signage using a flat display, a projector, etc. is installed in a place where people pass, and stereoscopic video is displayed It is also expected to do. The range in which an integral 3D image can be viewed correctly is called a viewing zone. The viewing zone corresponds to the angle at which the element image is projected by the element lens, and relates to the focal length of the element lens and the element image size. When an observer crosses the front of a monitor displaying an integral three-dimensional image, a double image may be seen when the monitor is viewed (see FIG. 6).

  Here, two possible causes of the double image are described with reference to FIGS. 6 and 7. FIG. 6 is a conceptual view showing an example of how a three-dimensional image looks, and FIG. 7 is a plan view schematically showing a state where a boundary of a viewing area is located between the right eye and the left eye. Some assumptions of the explanatory diagram of FIG. 7 will be described.

  On the monitor (display panel) 70, for example, portions of eight element images are displayed in the horizontal direction (vertical direction in FIG. 7). A lens array 80 is disposed in front of the monitor 70 (right in FIG. 7) at a predetermined distance. Here, a portion where, for example, eight element lenses 81 are arranged at a predetermined lens pitch in the horizontal direction is described. Further, in one viewing area, as one example, eleven viewpoint videos of viewpoint 1 to viewpoint 11 can be continuously observed from the left (lower in FIG. 7) in the horizontal direction. Adjacent viewing zones exist on the left and right of one viewing zone. In the illustrated state, the position of the left eye of the observer is in the vicinity of the viewpoint 11 in the viewing area, and the position of the right eye of the observer is in the vicinity of the viewing point 1 in the viewing area to its right. That is, the boundary between the two viewing areas is located between the right eye and the left eye of the observer. Note that the horizontal relationship for the observer regarding the element image 71, the element lens 81, and the viewing area is also established in the vertical direction for the observer (the direction perpendicular to the paper surface).

  The first cause of double image formation is that the boundary of the viewing area is located between the right eye and the left eye as shown in FIG. 6 and FIG. The observer sees a double image because an image of the area is observed. That is, as shown in FIG. 6, since two images 67 presented across the boundary of the viewing area are simultaneously observed, the observer sees as a double image 68. Since the viewing zone is limited by the monitor that displays the stereoscopic image and the lens structure, for example, when the observer crosses in front of the digital signage monitor, it passes through the boundary of the viewing zone of the integral stereogram. Therefore, when the monitor is viewed just when the observer's viewpoint is at the boundary of the viewing area, images of different viewing areas are observed by the right eye and the left eye, and a double image can be seen. The more different the images in the right and left eyes are, the larger the degree to which the double image looks double.

  Next, another cause will be described with reference to FIG. Some assumptions of the explanatory diagram of FIG. 10 (a) will be described. Here, a virtual space (CG space) is assumed. A lens array 180 is disposed in this virtual space. Here, a portion in which, for example, two element lenses 181 are arranged at a predetermined lens pitch in the horizontal direction (vertical direction in FIG. 10A) is described. The subject 182 is created by three-dimensional CG, and pixels forming the surface of the subject 182 are given predetermined pixel values (luminance values). In this example, the viewpoint of the observer (not shown) is located on the left side in FIG. 10A, and the element image is located on the right side of the element lens 181. Although the relationship in the horizontal direction holds in the vertical direction as well, the following description will be made using only the relationship in the horizontal direction.

In integral image generation, if the lens pitch and the pixel pitch of the monitor do not have an integer ratio, the lens pitch and the pixel pitch of the monitor have an integer ratio in the process of element image group generation processing A large image called a normalized element image is temporarily generated on the virtual space. The normalized element image is described in detail in, for example, Japanese Patent Application Laid-Open No. 2012-84105, and thus the description thereof is omitted here.
The element image group finally displayed on the monitor is obtained by reducing the normalized element image. Then, in order to reduce the size, the pixel part at the end of the element image constituting the image at the boundary of the viewing area is interpolated with the pixel at the opposite end of the element image adjacent to the right, for example Applied.

  Specifically, for example, the element image generated via the element lens 181 positioned below in FIG. 10A is represented by 11 pixels, that is, irradiation destinations of the light rays 1a to 11a. Similarly, an element image generated via the element lens 181 located at the top in FIG. 10A is represented by 11 pixels, that is, irradiation destinations of the light rays 1 b to 11 b. Then, in order to reduce the normalized element image, interpolation processing is performed between the pixel at the end of one element image (the end of the light ray 1a) and the pixel at the end of the other element image (the end of the light 11b) When applied, it will include both the component of the image obtained from the light ray 1a shown in FIG. 10A and the component of the image obtained from the light ray 11b. Therefore, a double image may be generated.

  In addition, depending on the relationship between the lens pitch and the pixel pitch of the monitor, the viewpoint may switch as if wiped at the boundary of the viewing area. Since two viewpoint videos are simultaneously displayed, the stereoscopic image at the time of this wipe generation causes a displacement in the stereoscopic image at the wipe portion, and may appear as a double image depending on the display content.

  A camera is used in place of the observer (right eye and left eye) in FIG. 7 and a state in which the viewpoint image obtained by this camera is switched discontinuously at the boundary of the viewing area will be described with reference to FIGS. 7 and 8. In FIG. 8, the horizontal axis represents the observation position and corresponds to the explanatory view of FIG. 7. The left side of the horizontal axis indicates that the observation position is also left, and the right side of the horizontal axis indicates that the observation position is also right. It shows. The vertical axis indicates video (viewpoint video n; n = 1 to 11) obtained by the camera corresponding to the viewpoint 1 to the viewpoint 11, respectively. That is, FIG. 8 shows the switching of the viewpoint image displayed when the camera facing the integral display (the monitor 70 in FIG. 7, the lens array 80) is parallel to the display surface and moved from left to right. It is shown in the graph.

  In FIG. 8, observation positions corresponding to about four viewing areas are displayed on the horizontal axis. For example, when the viewing angle of the display is about 10 to 30 °, when the observer (here, a camera) moves in front of the display, boundaries of the viewing area occur several times. As shown in FIG. 8, the image is switched discontinuously at the boundary of the viewing area.

Here, that the video is continuous means, for example, the relationship between the viewpoint video 6 and the viewpoint video 5 immediately adjacent to the left thereof, or the relationship between the viewpoint video 6 and the viewpoint video 7 immediately adjacent to the right thereof. Further, it can be said that the relationship between the viewpoint video 6 and the viewpoint video 4 skipped to the left, for example, and the relationship between the viewpoint video 6 and the viewpoint video 8 skipped to the right two, for example, are substantially continuous.
Then, at both ends of the viewing area, two viewpoint videos are continuous only on one side. In the case of the example of FIG. 8, for example, it is said that only the relationship between the viewpoint video 1 and the viewpoint video 2 immediately adjacent on the right is continuous, and the relationship between the viewpoint video 11 only immediately adjacent to the viewpoint video 10 immediately adjacent on the left is continuous. It is said that there is. Therefore, in the case of the example of FIG. 8, observation of the viewpoint video 1 immediately to the right of the viewpoint video 11 at the boundary of the viewing zone means that the viewpoint video is switched discontinuously.

  In short, in the case of a display that demonstrates a display that floats integral three-dimensional content suitable for exhibition in an exhibition or the like, or in the case of content that does not require much motion parallax representation to bring out the original attraction of the content. Conventionally, the relationship between the observation position and the viewpoint image may be discontinuous at the boundary of the viewing area as shown in FIG.

[2-1. Configuration of Element Image Group Generation Device]
In the element image group generation apparatus 10A according to the second embodiment shown in FIG. 9, the same components as those of the element image group generation apparatus 10 shown in FIG.
The element image group generation apparatus 10A, as shown in FIG. 9, includes an input interface 11, a lens array information storage unit 12, a viewpoint synthesis processing unit 13, a multiview video correction unit 14, and a multiview video recording and transfer unit An image switching unit 16A, an element image generation unit 17, an output interface 18, a three-dimensional model generation unit 19, and an oblique projection processing unit 20 are provided.

  The three-dimensional model generation means 19 receives an input signal of the three-dimensional scanner or an input video V composed of multi-view video which is a set of three or more multi-view video and receives the three-dimensional model on CG. To generate The method of generating the three-dimensional model is not particularly limited in the present invention, and thus the description thereof is omitted here. In addition, for example, it describes in detail in Unexamined-Japanese-Patent No. 2012-84105 and the prior literature of the said gazette description.

In the present embodiment, the three-dimensional model generation unit 19 includes a second determination unit 191.
The second determination means 191 determines whether or not an occlusion such as overlapping of objects has occurred in the generated three-dimensional model. The second determination means 191 notifies the multi-viewpoint video recording and transfer means 15 whether or not an occlusion has occurred.
Although the determination method here is not particularly limited, for example, the shape information of the area of the subject (three-dimensional CG) displayed in the previous frame and the shape information of the area of the subject (three-dimensional CG) at the current time In comparison, when the amount of change becomes larger than a preset threshold value, it is determined that an occlusion has occurred, and in other cases, it may be determined that an occlusion has not occurred. Here, as the shape information of the area of the subject (three-dimensional CG), for example, the area of a circumscribed rectangle that encloses the area of the subject (three-dimensional CG) can be used.

  In the present embodiment, the correction unit 153 of the multiview video record transfer unit 15 has the following function (the function of the second correction unit) in addition to the function (the function of the first correction unit) described in the first embodiment. . That is, when the second judging means 191 judges that the occlusion is occurring, the correcting means 153 projects the three-dimensional model in which the occlusion is occurring with respect to the viewpoint image in which the occlusion is occurring. The image data of the temporally adjacent frame or the spatially adjacent image data is read out from the recording buffer 151 and used to correct the viewpoint video in which the occlusion occurs.

  Since the oblique projection processing means 20 uses the image acquired by oblique projection in the integral type element image generation method, based on the lens array information, using the parallel projection method, the three-dimensional model is generated from an arbitrary viewpoint. It is projected to generate a multi-view image.

  Here, a method of generating an element image by oblique projection will be briefly described with reference to FIG. 10 (b). The premise of the explanatory view of FIG. 10B is substantially the same as that of FIG. 10A, and when arranging the lens array 180 as a reference in the virtual space, the three-dimensional model of the object is in the vicinity of the lens array 180. Place on In FIG. 10B, the optical axis of the element lens 181 is indicated by a broken line.

  In this example, the viewpoint of the observer (not shown) is located on the left side in FIG. 10 (b), and the subject 182 which is a three-dimensional model is in front of the lens array 180 (FIG. 10 (b)). The subject 183, which is another three-dimensional model, is disposed so as to straddle the front and the back of the lens array 180 (right in FIG. 10B). The three-dimensional model may be disposed at the back of the lens array 180.

The display surface (virtual display) 184 of the element image group is placed at a position separated from the lens array 180 by the focal length of the lens array 180 (to the right in FIG. 10B).
For example, in the method of generating an element image by oblique projection, as shown in FIG. 10B, the direction of a ray passing through the center of the element lens 181, ie, the direction inclined by the angle θ from the direction of the optical axis of the element lens 181. The pixel position in the element image can be determined from the direction of the ray, and the pixel value can be determined from the intensity of the ray. Further, by repeating the oblique projection method shown in FIG. 10B for each light ray angle as shown by light rays 1a to 11a shown in FIG. 10A, all pixel values of the element image can be obtained. .

  In other words, from the side of the pixel on the display surface 184 of the element image group, a straight line connecting the pixel and the center (optical principal point) of the element lens 181 closest to the pixel (light ray in FIG. 10A) The pixel value of the intersection closest to the viewpoint is given to the pixel of the element image among the intersections at which the straight line and the target three-dimensional model intersect. And if this process is performed about all the pixels, an element image group can be produced | generated.

The multi-viewpoint video recording and transferring means 15 also has a function of temporarily recording a multi-viewpoint video generated by projecting a three-dimensional model in the recording buffer 151.
The image switching unit 16A is a multi-view image generated by projecting a three-dimensional model, a multi-view image generated by the viewpoint synthesis processing unit 13, a multi-view image generated by the correction of the multi-view image correction unit 14, and recording. Selecting and outputting any one of the multi-view videos recorded in the buffer 151 is repeated in time division. These video switching may be switched to a predetermined timing before operation or the like by an instruction operation by the operator of the element image group generation device 10A.

  When selecting the multi-view image transferred from the multi-view image record transfer unit 15, that is, when outputting the multi-view image to the element image generation unit 17, the image switching unit 16A performs the following (first case) , (Second case) or (third case) to select a multi-viewpoint image. Here, with reference to FIG. 3 and FIG. 4 (refer to FIG. 9 as needed), the position of the pixel of the element image will be described while being additionally illustrated.

(First case)
When the element image generation means 17 at the subsequent stage generates the element image at the upper left of FIG. 4, the image switching means 16A generates the left end (eg, pixel position 11 of FIG. 3) and the right end (eg, FIG. 3) of the element image. The multi-viewpoint video is selected and an assigned multi-viewpoint image group is generated on the memory so that each element value of each of the pixels located at the pixel position 31) becomes an element image having a continuous value (see FIG. 3).

(Second case)
When the element image generation means 17 in the subsequent stage generates the element image at the upper left of FIG. 4, the image switching means 16A generates the upper end (for example, the pixel position 11 of FIG. 3) and the lower end (for example, FIG. 3) of the element image. The multi-viewpoint video is selected and an assigned multi-viewpoint image group is generated on the memory so that each element value of each of the pixels located at the pixel position 71) becomes an element image having a continuity value (see FIG. 3).

(Third case)
When the element image generation means 17 in the subsequent stage generates, for example, the upper left element image in FIG. The pixel values of the pixels respectively located at the pixel position 11) in FIG. 3, the lower end (for example, the pixel position 71 in FIG. 3), and the right end (for example, the pixel position 31 in FIG. 3) The multi-viewpoint video is selected to generate an assigned multi-viewpoint image group on the memory so as to obtain an element image having the above (see FIG. 3).

  These functions of the image switching unit 16A are functions added to the configuration of the element image group generation device 10 according to the first embodiment to display a more natural three-dimensional image. A specific example of this additional function will be described later.

[2-2. Overall Flow of Processing by Element Image Group Generator]
The entire flow of processing by the element image group generation device 10A according to the second embodiment will be described with reference to FIG. 9 except for the duplication with the first embodiment. In the first embodiment, an example of generating an element image from a multi-view video has been described, but here, a process of generating an element image from a three-dimensional model will be described.

  The three-dimensional model generation unit 19 of the element image group generation device 10A generates a three-dimensional model when the input signal S of the three-dimensional scanner and the input video V such as multi-view video are input. Then, the oblique projection processing means 20 receives information such as the lens pitch from the lens array information storage means 12 and performs oblique projection from a certain viewpoint on the generated three-dimensional model, and a viewpoint image obtained by oblique projection Are output to the video switching means 16 and the multi-view video record transfer means 15.

  Then, the recording buffer 151 of the multiview video record transfer unit 15 temporarily records a multiview video which is a set of viewpoint videos. Then, if there is no occurrence of occlusion in the three-dimensional model, the image switching means 16A outputs the multi-viewpoint image as it is to the element image generation means 17. On the other hand, when an occlusion occurs, the multi-viewpoint video recording and transferring means 15 corrects the video compared with the viewpoint image etc. from the adjacent frame called from the recording buffer 151 and outputs it to the element image generating means 17 Do.

[2-3. Processing of additional functions of image switching means]
The processing of the additional function of the image switching means will be described below with reference to FIGS. 10 to 12 (refer to FIGS. 7 and 8 as appropriate). In the method of generating an element image by oblique projection, as shown in FIG. 10B, attention is focused on the direction of a ray passing through the center of the element lens 181, that is, the direction inclined by the angle θ from the direction of the optical axis of the element lens 181. doing. Here, as shown in FIG. 10A, the light ray 1a and the light ray 1b passing through the element lens 181 next to the light ray 1a have the same direction, and can be considered as a part when the viewpoint 1 is parallel projected. . Similarly, the light ray 11a and the light ray 11b can be considered as a part when the viewpoint 11 is projected in parallel.

  On the other hand, the element image group generation apparatus 10A generates a multi-viewpoint image by projecting a three-dimensional model from an arbitrary viewpoint (for example, the viewpoint 1 and the viewpoint 11) by using oblique projection processing means 20 using parallel projection. doing. Therefore, an image of parallel light from the viewpoint 1 to the viewpoint 11 in this example is temporarily recorded in the recording buffer 151 of the multi-viewpoint image recording and transferring means 15. Therefore, the video switching unit 16A selects the multi-viewpoint image as necessary from the images of parallel light from the viewpoint 1 to the viewpoint 11 recorded in the recording buffer 151, that is, the multi-viewpoint image group of 11 viewpoints. It is possible to rearrange and allocate the arrangement on the memory.

  In other words, the video switching unit 16A can perform processing of replacing video so that the multi-viewpoint images recorded in the recording buffer 151 do not become discontinuous. FIG. 11 is a graph in the case of replacing the video so as to give continuity. The view of the graph of FIG. 11 is similar to that of the graph of FIG. In the case of the graph of FIG. 8 as a comparative example, as described above, a total of 11 multi-viewpoint images from viewpoint 1 to viewpoint 11 corresponding to one section of the horizontal viewing area in FIG. Represents a graph that assumes that. On the other hand, the video switching means 16A, as shown in FIG. 11, corresponds to one segment of the viewing area, “multi-viewpoint image“ viewpoint 1 → viewpoint 2 → viewpoint 3 → viewpoint 4 → viewpoint 5 → viewpoint 6 → The element image is generated by reading as “viewpoint 7 → viewpoint 8 → viewpoint 6 → viewpoint 4 → viewpoint 2”.

  That is, although not all viewpoints corresponding to one section of the viewing area are used, a total of 11 readings are performed so that the multi-viewpoint image is continuous or substantially continuous. As a result, in some sections, for example, as in “view point 8 → view point 6”, it may be reverse vision. However, instead of that, the parallax of the image entering the right eye and the left eye at the boundary of the viewing area is “viewpoint 11 → viewpoint 1” in the case of FIG. As 2 → viewpoint 1 ”, there is an effect of reducing parallax. In addition, according to the processing of the video switching means 16A, it is possible to reduce the difference between the pixel values at both ends of the element image at the time of generating the element image group by reducing from the normalized element image. The image can be reduced.

  FIG. 12 is a graph in the case of replacing the image for the purpose of completely erasing the double image at the boundary of the viewing zone. The view of the graph of FIG. 12 is similar to that of the graph of FIG. In this case, as shown in FIG. 12, the video switching means 16A displays the multi-viewpoint image as “viewpoint 4 → viewpoint 5 → viewpoint 6 → viewpoint 7 → viewpoint 8 → viewpoint 9 → viewpoint 8 → viewpoint 7 → viewpoint 6 → viewpoint 6 → viewpoint 5 → view 4 ”to generate an element image. That is, although not all 11 viewpoints corresponding to one section of the viewing area are used, a total of 11 readings are performed so that multi-view images are continuous.

  Further, in the graph of FIG. 12, when focusing on the boundary of the viewing area, the same viewpoint video 4 is arranged next to the viewpoint video 4. Such an arrangement can also be regarded as "continuous" of the viewpoint video. The image switching means 16A selects the multi-view image in this manner and generates an assigned multi-view image group on the memory, for example, pixels arranged at both ends among the pixel groups constituting the element image of FIG. The pixel values of (hereinafter, simply referred to as pixel values at both ends of the element image) are generated using the same viewpoint video (here, viewpoint video 4).

  In the case of FIG. 12 as well as FIG. 11, there is a demerit that the reverse vision is in a part of the area as in the case of FIG. Furthermore, although the second cause of double image generation is caused by reduction from a normalized element image to an element image group, in the example of FIG. 12, the pixel values at both ends of the element image are made the same value. It will be possible to erase the double image.

  As described above, according to the element image group generation apparatus 10A according to the present embodiment, switching from, for example, the viewpoint 1 to the viewpoint 11 is discontinuously at the boundary of the viewing area in the graph of FIG. For example, the video from the viewpoint 1 to the viewpoint 11 is temporarily recorded in the recording buffer 151 of the multi-viewpoint video recording and transfer means 15 so as to avoid the video so as to continuously switch the viewpoint video at the boundary of the viewing area. Perform replacement process. This reduces the double image at the boundary of the viewing zone.

  Similarly, according to the element image group generation apparatus 10A, for example, images of parallel light from the viewpoint 1 to the viewpoint 11 are temporarily recorded in the recording buffer 151 of the multi-viewpoint image recording and transfer means 15, and the images of parallel light are appropriately recorded. The replacement also reduces the double image produced by the second cause “generate a normalized element image because the lens pitch and monitor pixel pitch are not an integer ratio and further reduce the normalized element image”. can do.

Further, according to the element image group generation device 10A, the replacement process using the recording buffer 151 as described above can similarly reduce double images due to displacement of a solid image generated in the wipe portion at the time of wipe generation. .
Furthermore, according to the element image group generation device 10A, even when occlusion occurs in the three-dimensional model, the viewpoint video can be corrected and generated.

Third Embodiment
[3-1. Broadcast communication cooperation system]
A broadcast communication cooperation system including a digital broadcast receiving apparatus according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 13, the broadcast-communication collaboration system 1 includes a service providing system 2 and digital broadcast reception devices 3 (3a, 3b). The viewer views the program content by the digital broadcast receiving apparatus 3. Hereinafter, the term "user" refers to a viewer.

  The service providing system 2 is a system for providing the viewer with stereoscopic video content using the communication network N as a predetermined service, and supplements the broadcast content with the digital broadcast transmitting apparatus 4 used in the broadcast station to broadcast the video content. And a web server 5 for providing communication content.

  The digital broadcast transmission device 4 and the Web server 5 on the broadcast station side and the digital broadcast reception device 3 on the viewer side are configured by functional blocks as shown in FIG. 14, for example. The details of the configuration of each device will be described below.

[3-2. Digital broadcast transmitter]
The digital broadcast transmission device 4 broadcasts program content, and provides stereoscopic video content to a viewer as one service. The digital broadcast transmission device 4 includes a content storage unit 41, a broadcast control unit 42, and a broadcast transmission unit 43.

  The content storage unit 41 stores various contents such as program contents of digital broadcast and data broadcast contents, and is a storage device such as a hard disk. The content storage unit 41 stores, as program content, for example, multi-view video (video content) having horizontal parallax and vertical parallax, or multi-view video (video content) which is a set of viewpoint videos only with horizontal parallax. There is.

The broadcast control unit 42 reads program content, data broadcast content, and the like from the content storage unit 41 and outputs the read content to the broadcast transmission unit 43. The broadcast control means 42 performs various controls such as scrambling of content and encryption of key information corresponding to the conditional access system of digital broadcast, although detailed description will be omitted.
The broadcast sending means 43 multiplexes the program content output from the broadcast control means 42, the data of the data broadcast and the like, and transmits (distributes) the data by a data carousel.

[3-3. Web server]
The web server 5 is prepared by a service provider, and is a server such as CGI (Common Gateway Interface) or PHP (Hypertext Preprocessor). In the present embodiment, a broadcast station (broadcaster) also serves as a service provider.

  The Web server 5 includes, for example, an arithmetic device such as a central processing unit (CPU), a storage device (storage means) such as a memory or a hard disk, and an interface device that transmits and receives various information with the outside. The hardware resource is implemented by being controlled by a program, and as shown in FIG. 14, the content storage unit 51 and the content distribution unit 52 are provided.

The content storage unit 51 stores communication content, and includes a storage device such as a memory or a hard disk. Here, when a multi-view video (video content) which is a set of viewpoint video with only horizontal parallax by broadcast wave W is provided, communication content is a multi-view video which is a set of viewpoint video with vertical parallax of the video content. Remember as.
The content distribution means 52 distributes communication content to the digital broadcast receiving apparatus 3a connected to the communication network N. The video content transmitted by broadcast wave W and the video content distributed by communication network N are synchronized and reproduced by using a time based on Coordinated Universal Time (UTC). It is possible. Further, the content distribution means 52 has a function of distributing video content at an arbitrary timing by receiving a request by the operation of the operator of the digital broadcast receiving apparatus 3.

[3-4. Digital broadcast receiver]
The digital broadcast receiving apparatus 3 receives program content and data broadcast by broadcast waves W and the like. Here, the digital broadcast receiving apparatus 3 includes a broadcast receiving unit 31, a user operation unit 32, a communication unit 33, a content storage unit 34, a program storage unit 35, a reception control unit 36, a monitor 70, and a lens. And an array 80. Then, the digital broadcast receiving apparatus 3 can receive digital broadcast by mounting the individual information storage card 37 which is a security module. The individual information storage card 37 is, for example, a B-CAS card.

The broadcast receiving means 31 is composed of a tuner or the like for receiving the digital broadcast sent from the digital broadcast transmitter 4 and outputs the digital signal received when the individual information storage card 37 is mounted to the reception control means 36.
The user operation means 32 outputs a command designated by the user's operation to the reception control means 36, and is, for example, a remote control device.

  The communication unit 33 transmits and receives information to and from the outside via the communication network N. The communication means 33 uses the broadcast wave W to provide a multi-viewpoint video (video content) which is a set of viewpoint videos of only horizontal parallax, and provides the Web with a multi-viewpoint video which is a set of viewpoint video of vertical parallax of the video content. It receives from the server 5 and outputs it to the element image group generation device 10B.

The content storage unit 34 stores content such as a digital broadcast program received by the broadcast reception unit 31 and is configured by a hard disk or the like.
The program storage unit 35 stores operation programs of the reception control unit 36 and the like, processing results thereof, and the like, and is configured by a memory, a hard disk, and the like.

  The reception control means 36 controls reproduction and display of the digital signal received by the broadcast reception means 31 and the communication means 33. Here, the reception control unit 36 includes a content reproduction control unit 361, a display control unit 362, and an element image group generation device 10B.

  The content reproduction control means 361, for example, separates encryption key information and content from multiplexed digital signals received by the broadcast wave W, decrypts the key information, and reproduces the content by descramble the content. It controls.

  The display control means 362 causes the monitor 70 to display the content reproduced by the content reproduction control means 361. The display control means 362 analyzes BML, which is an electronic document for data broadcasting, and performs display control. Further, the display control means 362 may analyze the HTML 5 of the hybrid cast and display additional information of the program.

The element image group generation apparatus 10B is the same as the element image group generation apparatus 10A in FIG. 9 except for the configuration of the display switching unit 21. Therefore, the detailed description is omitted. The lens array information storage unit 12 and the recording buffer 151 in the element image group generation apparatus 10B are also used as the program storage unit 35, for example.
The digital broadcast receiving apparatus 3 displays an integral-type three-dimensional image from the element image group generated by the element image group generating apparatus 10B.

  The display switching means 21 switches between the display of an integral 3D image without vertical parallax and the display of an integral 3D image having both horizontal and vertical parallax. Here, a mode for displaying an integral stereo image without vertical parallax is referred to as a first mode, and a mode for displaying an integral stereo image having both horizontal parallax and vertical parallax is referred to as a second mode.

  In the first mode, when the digital broadcast receiving apparatus 3 receives the multi-view video of only horizontal parallax from the broadcast wave W by the broadcast receiving unit 31, an element image is generated based on the multi-view video of only the horizontal parallax received by the broadcast. The integral-type three-dimensional image without vertical parallax generated by the group generation device 10B is displayed.

  Further, in the second mode, when the digital broadcast receiving apparatus 3 receives the multiview video of only horizontal parallax from the broadcast wave W by the broadcast receiving unit 31, the multiview video other than that is transmitted by the communication unit 33 through the communication network N. Integral method having both horizontal parallax and vertical parallax generated by the element image group generation device 10B using the multi-view image received through the network and the multi-view image received from the broadcast wave W in combination Display a stereoscopic image of

  The reception control means 36 can be realized by operating a computer built in the digital broadcast receiving apparatus 3 with a receiver program that causes the computer to function as the above-described means. The receiver program is provided from the digital broadcast transmitter 4 via data broadcast. The element image group generation program, which is a part of the receiver program, is a program for causing a computer to function as the element image group generation device 10B.

  The monitor (display panel) 70 displays a digital broadcast program and data broadcast decoded as a result of processing by the reception control means 36 on a screen, and is, for example, a liquid crystal display (LCD) or the like. The lens array 80 is not particularly limited as long as it has a lens group applicable to the integral method.

[3-5. Operation of Element Image Group Generator]
In the first mode, the element image group generation device 10B performs the same operation as the element image group generation device 10 of the first embodiment.
In the second mode, the element image group generation device 10B outputs the communication content (multi-view video which is a set of viewpoint videos having vertical parallax) received by the communication unit 33 to the multi-view video record transfer unit 15. If necessary, the respective processing is performed via the viewpoint combination processing unit 13 and / or the multi-viewpoint image correction unit 14. Then, the elemental image group generation device 10B is configured to record the viewpoint video with vertical parallax recorded by the video switching unit 16 in the recording buffer 151 (program storage unit 35) and the horizontal parallax recorded in the recording buffer 151 in the same manner. For example, it is assigned to the multi-viewpoint image arrangement area 61 of FIG. Then, the element image group generation device 10B generates an element image from the multi-viewpoint image with horizontal parallax and vertical parallax by the element image generation unit 17.

  As described above, according to the present embodiment, it is possible to efficiently increase the number of video contents that can be displayed on the integral display required for future television broadcasting.

As mentioned above, although the element image group production | generation apparatus which concerns on this invention based on embodiment was demonstrated, this invention is not limited to these. For example, although the element image is generated based on an image acquired by oblique projection, for example, as the integral element image generation method in the embodiment, the projection method is not limited to oblique projection.
Further, the means for performing dropout determination and occlusion occurrence determination of the viewpoint video is not limited to being provided at the position of the block configuration of the device according to the embodiment, and in the second embodiment, two means are integrated to perform It may be arranged as one block configuration.
In the second embodiment, the function of the correction unit 153 is divided into a first correction unit that corrects when there is a loss of a viewpoint video, and a second correction unit that corrects when an occlusion occurs. You may do so.

Further, in the description of FIG. 2, an integral-type multi-viewpoint image is described by capturing an image of 3 × 3 viewpoints with a camera array, but the number of viewpoints is not limited to this. It may be a viewpoint or more.
In the lens array used for integral-type three-dimensional display, the arrangement of the element lenses is not limited to the square array arranged vertically and horizontally, and the element lenses are shifted in the vertical direction and the horizontal direction with respect to the arrangement of one row of element lenses. It may be arranged to be in piles.

Although the element image of FIG. 4 is square, it may be another shape such as a circle.
Further, the entire lens array may be arranged to be inclined such that the lens arrangement direction is oriented in a direction inclined to the arrangement direction of the element images.
Further, the lens array is not limited to the convex lens group arranged in a plane as shown in FIG. 7, and another lens group or a pinhole group may be disposed.

  The digital broadcast receiving apparatus 3 according to the third embodiment is to switch between display of an integral 3D image without vertical parallax and display of an integral 3D image having both horizontal and vertical parallax. Although the display switching means 21 is provided, the digital broadcast receiving apparatus 3 may have a configuration to automatically determine, or the switching flag is superimposed on the data received from the communication network N and this flag It may be switched by.

The digital broadcast receiving apparatus 3 according to the third embodiment includes information acquired from the broadcast wave W and information acquired from the communication network N in order to display an integral-type three-dimensional image having both horizontal parallax and vertical parallax. However, all information necessary for the display may be obtained from the communication network N.
Similarly, all information necessary for displaying an integral stereo image having both horizontal parallax and vertical parallax may be acquired from the broadcast wave W. In this case, it is not necessary to connect to the communication network N such as the Internet as in the digital broadcast receiver 3b of FIG.

DESCRIPTION OF SYMBOLS 1 broadcast communication cooperation system 2 service provision system 3, 3a, 3b digital broadcast receiver 4 digital broadcast transmitter 5 Web server 10, 10A, 10B element image group generator 11 input interface 12 lens array information storage means 13 viewpoint synthesis process Means 14 multi-viewpoint image correction means 141 oblique projection processing means 142 first determination means 15 multi-view image recording / transfer means 151 recording buffer 152 image output means 153 correction means (first correction means, second correction means)
16 image switching means 17 element image generation means 18 output interface 19 three-dimensional model generation means 191 second determination means 20 oblique projection processing means 21 display switching means 31 broadcast receiving means 32 user operation means 33 communication means 34 content storage means 35 program Storage means 36 Reception control means 361 Content reproduction control means 362 Display control means 37 Individual information storage card 41 Content storage means 42 Broadcast control means 43 Broadcast sending means 51 Content storage means 52 Content distribution means 61, 64 Multi-view image arrangement area 62, 65, 66 multi-viewpoint image group 63 element image group 70 monitor (display panel)
71 Elemental Image 80, 180 Lens Array 81, 181 Element Lens 182, 183 Object (3D CG)
184 Display surface of element image group (virtual display)

Claims (7)

An element image group generation apparatus for generating an element image group which is a plurality of integral element images of an integral type displayed on a display panel to present a stereoscopic image through a lens array,
When an image composed of viewpoint images of a plurality of viewpoints corresponding to only the horizontal parallax between viewpoints is input, the horizontal parallax is input based on lens array information including information on the lens pitch of the lens array. Viewpoint synthesis processing means for synthesizing a viewpoint from a plurality of viewpoint videos held and generating a multi-view image composed of viewpoint videos in which the number of viewpoints is increased compared to the input video;
A multi-viewpoint image correction unit that corrects the multi-viewpoint image generated by the view point synthesis processing unit into an image of a projection method suitable for an integral-type element image generation method;
A predetermined multi-viewpoint image is selected, and a multi-viewpoint image which is a multi-viewpoint image per unit time is allocated in the horizontal direction and the vertical direction of the multi-viewpoint image arrangement area defined on the memory Video switching means for repeatedly performing a process of generating a multi-viewpoint image group in time division;
Element image generation means for generating the element image group by repeating the process of generating an element image from the multi-viewpoint image group by the integral type element image generation method;
A multi-view video recording and transfer unit that records a multi-view video generated by the correction of the multi-view video correction unit in a recording buffer and transfers the multi-view video from the recording buffer to the video switching unit;
The video switching means is
Based on the multi-viewpoint video transferred from the multi-viewpoint video recording and transfer means, in the horizontal direction of the multi-viewpoint image arrangement area, the view point video of the horizontal component generated by the view synthesis processing means is allocated. An element image group generating apparatus characterized by performing processing of generating the multi-viewpoint image group by repeatedly assigning viewpoint images at the same horizontal position in the vertical direction of the multi-viewpoint image arrangement area. The video switching means is
When selecting the multi-view video transferred from the multi-view video record transfer unit,
Pixel values of pixels respectively located at left and right ends of the element image, or pixel values of pixels respectively located at upper and lower ends, or pixels respectively located at any three ends of upper, lower, left, and right 2. The element according to claim 1, wherein the multi-viewpoint image is selected and output to the element image generation unit so that each pixel value of the element image becomes an element image having a continuous value. Image group generator. The element image group generation is performed by comparing the multi-viewpoint image of interest with the image data of the frame temporally adjacent to the multi-view image or the image data spatially adjacent with reference to the recording buffer. A first determination unit that determines whether or not the video composed of the viewpoint video input for each frame to the device has a drop in the viewpoint video that constitutes the video;
When there is a loss of the viewpoint video, image data of a frame temporally adjacent to the missing portion or image data spatially adjacent is read out from the recording buffer and used to correct the image of the missing portion. First correction means,
The element image group generation apparatus according to claim 1 or 2, further comprising: 3D model generation means for generating a 3D model on CG by receiving an input signal of a 3D scanner or an input video composed of multiview video which is a set of viewpoint video of 3 or more multiple viewpoints respectively;
Projection processing means for projecting the three-dimensional model from an arbitrary viewpoint using a parallel projection method based on the lens array information to generate a multi-viewpoint image;
The multi-viewpoint video recording and transfer unit records, in the recording buffer, a multi-viewpoint video generated by projecting the three-dimensional model.
The video switching means is
A multi-view image generated by projecting the three-dimensional model, a multi-view image generated by the view synthesis processing unit, a multi-view image generated by the correction of the multi-view image correction unit, and the recording buffer The element image group generation apparatus according to any one of claims 1 to 3, wherein selecting and outputting any one of the multi-viewpoint images is repeatedly performed in time division. A second determination unit that determines whether occlusion has occurred in the generated three-dimensional model;
The recording buffer is image data of a frame temporally adjacent to a viewpoint image temporally adjacent to a viewpoint image in which the occlusion is generated by projecting the three-dimensional model in which the occlusion is generated, or the image buffer as the recording buffer A second correction unit that corrects the viewpoint image in which the occlusion is generated by reading out from the
The element image group generation apparatus according to claim 4, comprising:   An element image group generation program for causing a computer to function as the element image group generation apparatus according to any one of claims 1 to 5. The element image group generation apparatus according to any one of claims 1 to 5, the display panel, the lens array, a broadcast receiving unit for receiving a broadcast wave, and an external device via a communication network. And a communication means for transmitting and receiving information between each other, wherein the digital broadcast receiving apparatus displays an integral-type three-dimensional image from the element images generated by the element image group generating apparatus,
When the multi-view video of only horizontal parallax is received from the broadcast wave by the broadcast receiving means,
Display of an integral-type three-dimensional image without vertical parallax generated based on the multi-view image of only horizontal parallax received in the broadcast;
The multi-view image received through the communication network by the communication means and multi-view image received from the broadcast wave is received from the multi-view image other than the multi-view image only horizontal parallax received by the broadcast wave Integral stereoscopic image display having both horizontal parallax and vertical parallax generated in combination;
A digital broadcast receiver characterized by comprising display switching means for switching the

Images