JP2021044659A - Encoding device, decoding device and program - Google Patents

Abstract

To provide an encoding device, a decoding device and a program that can perform compressing/encoding of a multi-viewpoint video in high encoding efficiency.SOLUTION: An encoding device comprises: a viewpoint thinning-out unit that performs viewpoint thinning-out processing of a multi-viewpoint image group; a multi-viewpoint image element image conversion unit that converts the viewpoint thinned-out multi-viewpoint image group to an encoding element image group; and an encoding unit that performs encoding processing of the encoding element image group. The decoding device comprises: a decoding unit that decodes image encoding data, and that creates the encoding element image group; an element image multi-viewpoint image conversion unit that converts the encoding element image group to the multi-viewpoint image group; a depth estimation unit that performs depth estimation of each multi-viewpoint image on the basis of the multi-viewpoint image group, and that creates a depth map; and a viewpoint interpolation processing unit that performs viewpoint interpolation between viewpoints of the multi-viewpoint image group on the basis of the multi-viewpoint image group and the created depth map.SELECTED DRAWING: Figure 1

Description

The present invention relates to a coding device, a decoding device, and a program, and more particularly, a multi-viewpoint image coding device, a decoding device, and a program necessary for displaying an integral 3D (three-dimensional) image and a free-viewpoint image. Regarding.

As a camera capable of capturing an element image group for displaying an integral 3D image, a light field camera in which a lens array is arranged in front of a sensor of an image sensor has been commercialized. However, in general, a light field camera aims at a refocus function after shooting. Therefore, when an integral 3D image is displayed using an image taken by a light field camera, the diameter of the main lens constituting the light field camera becomes a small value compared to the distance to the subject, so that the motion disparity is small. It is not possible to sufficiently reproduce the depth of a three-dimensional image. This problem can be theoretically solved by increasing the diameter of the main lens or shortening the distance between the camera and the subject, but it is not practical to solve the problem by these measures.

Therefore, it is considered to shoot a multi-viewpoint image by using a camera array in which ordinary cameras are arranged in a horizontal and vertical two-dimensional array. In this case, the element image group is generated by using the viewpoint interpolation processing from a plurality of images taken by the camera array to generate the viewpoint image between the cameras, and then from the image taken by the camera array and the viewpoint interpolation image. A process of converting into an element image group is performed (Patent Document 1). Here, it is known that the inter-camera distance of the camera array can be designed by the distance from the camera to the subject, the distance where viewpoint interpolation is practically possible, and the viewing range angle that can be reproduced by the display device. Further, in the viewpoint interpolation processing, a highly accurate interpolation image is generated by using a depth map (depth image) that relatively expresses the distance from the camera to the subject. Depth maps are generated by depth estimation using image processing technology or optical distance measurement using infrared rays. Increasing the accuracy of this depth map generation also improves the accuracy of viewpoint interpolation.

Regarding the display of an integral 3D image, the depth at which a three-dimensional image can be reproduced is related to the parallax between adjacent multi-viewpoint images, the focal length of the lens array, and the number of pixels of the element image. Among them, it is known that it is effective to increase the number of pixels of the element image in order to increase the depth at which the three-dimensional image can be reproduced. In this case, since the number of pixels of the element image is equal to the number of viewpoints of the multi-viewpoint image, a large number of viewpoints of the multi-viewpoint image to be encoded is required to generate a deep three-dimensional image. The amount of information for displaying a three-dimensional image is enormous.

In the transmission and recording of integral 3D video, a huge amount of information for displaying a three-dimensional video is encoded. In coding, a method of converting an element image group into a multi-viewpoint image group and then performing multi-viewpoint video coding, and a method of thinning out the converted multi-viewpoint video at the time of coding and interpolating the viewpoint at the time of decoding are known.

In the future, television broadcasting of integral 3D images is also envisioned. When performing television broadcasting of integral 3D video, it is required to ensure compatibility with conventional television broadcasting (hereinafter referred to as 2D (two-dimensional) broadcast video).

Japanese Unexamined Patent Publication No. 2016-158213

However, in the conventional multi-viewpoint video coding, when the parallax between the multi-viewpoint images to be coded becomes large, the accuracy of the viewpoint compensation prediction is lowered and the coding efficiency is deteriorated. Further, when the number of viewpoints of the multi-viewpoint video is increased and the coding is performed, the amount of information of the three-dimensional video to be encoded becomes large, and highly efficient compression cannot be performed.

In addition, regarding the simultaneous distribution of integral 3D video TV broadcasting and conventional 2D broadcast video, in the distribution using the Internet line, the transmitting side receives data in an image format suitable for each display terminal from the receiver. Simultaneous distribution is not a big problem because the requested data is transmitted after the request, but when transmission in broadcast waves is assumed, integral 3D video and 2D are used to ensure compatibility with conventional broadcasting. It is necessary to transmit both information to display the broadcast video for use. Therefore, in order to encode a huge amount of data obtained by adding a 2D broadcast video to an integral 3D video, it is required to perform coding with high coding efficiency.

Therefore, an object of the present invention made in view of the above problems is to provide a coding device, a decoding device, and a program capable of compressing and coding a multi-viewpoint video with high coding efficiency. is there.

In order to solve the above problems, the present invention converts a multi-viewpoint image into an element image group and performs a coding process on the coding side. The amount of information is reduced by compressing the element image group using a video coding method that has an intra-block copy function. Further, on the decoding side, the decoded element image group is converted into a multi-viewpoint image, and after the viewpoint interpolation processing is performed, the conversion to the element image group is performed. Further, pixels of a 2D broadcast image are introduced as a part of the element image. In the present specification, the "image" includes a moving image and may be a so-called "video".

In order to solve the above problems, the coding apparatus according to the present invention has a viewpoint thinning unit that performs viewpoint thinning processing on an input multi-view image group, and an element image for encoding the multi-view image group that has been thinned out. It is characterized by including a multi-viewpoint image element image conversion unit that converts into a group and a coding unit that encodes the coding element image group.

Further, it is desirable that the coding apparatus uses a coding tool having an intra-block copy function for the coding process.

Further, it is desirable that the coding apparatus makes the pixel size of the element image of the coding element image group equal to the coding block unit.

Further, a 2D (two-dimensional) image is further input to the coding device, and the multi-viewpoint image element image conversion unit receives pixels or pixels corresponding to the 2D image in each element image of the coding element image group. It is desirable to embed blocks.

In order to solve the above problems, the decoding apparatus according to the present invention decodes the input image coding data and creates a coding element image group, and a decoding unit, and the coding element image group is a multi-viewpoint image. Element image to be converted into a group Image multi-viewpoint image conversion unit, a depth estimation unit that estimates the depth of each multi-viewpoint image based on the multi-viewpoint image group and generates a depth map, and the multi-viewpoint image group are generated. It is characterized by including a viewpoint insertion processing unit that performs viewpoint insertion between viewpoints of the multi-view image group based on the depth map.

Further, it is desirable that the decoding device further includes a multi-viewpoint image element image conversion unit that converts the multi-viewpoint image group interpolated into the viewpoint into an element image group, and outputs the element image group.

Further, in the decoding device, a 2D (two-dimensional) image is embedded in the coding element image group, and the element image multi-viewpoint image conversion unit is used from each element image of the coding element image group. It is desirable to extract the pixels or pixel blocks of the 2D image, integrate them, and reproduce and output the 2D image.

A program according to the present invention for solving the above problems is characterized in that a computer functions as the coding device.

A program according to the present invention for solving the above problems is characterized in that a computer functions as the decoding device.

According to the coding device, the decoding device, and the program of the present invention, the multi-viewpoint video can be compressed and coded with high coding efficiency.

It is an example of the block diagram of the coding apparatus and decoding apparatus of 1st Embodiment. It is a figure explaining the conversion from the multi-viewpoint image group to the element image group for coding. It is a figure explaining the conversion from a multi-viewpoint image group to an element image group in a decoding apparatus. It is an example of the block diagram of the coding apparatus and decoding apparatus of the 2nd Embodiment. It is a figure explaining the conversion from a multi-viewpoint image group to a coding element image group in which a 2D image is embedded.

Hereinafter, embodiments of the present invention will be described.

(First Embodiment)
FIG. 1 shows an example of a block diagram of a coding device and a decoding device according to the first embodiment of the present invention. The coding device 10 and the decoding device 20 together constitute a coding / decoding system. The coding device 10 and the decoding device 20 may be connected by an arbitrary transmission line capable of information communication, and in this case, both function as the transmitting device 10 and the receiving device 20. As a transmission / reception method at this time, a broadcasting system, radio wave communication, a wired / wireless network, or the like can be used. Further, both may be independent devices, and data may be exchanged from the coding device 10 to the decoding device 20 using a recording medium or the like.

Hereinafter, each of the coding device 10 and the decoding device 20 will be described in detail.

[Coordinator]
The coding device 10 includes a viewpoint thinning unit 11, a multi-view image element image conversion unit 12, and a coding unit 13.

The input image is, for example, a group of multi-viewpoint images acquired by a multi-viewpoint camera in which 22 × 22 (= 484) cameras (for example, CMOS sensors) are arranged vertically and horizontally. Each of the images from one viewpoint is a color texture image. The input image is input to the viewpoint thinning unit 11.

The viewpoint thinning unit 11 performs a viewpoint thinning process for thinning out the viewpoints at equal intervals for the input multi-view image group. For example, the 22 × 22 viewpoint is thinned out to reduce the image to an 8 × 8 viewpoint image. The thinned out multi-viewpoint image group is output to the multi-viewpoint image element image conversion unit 12.

The multi-viewpoint image element image conversion unit 12 converts the input (thinned out) multi-viewpoint image group into a coding element image group. Here, the conversion to the element image group for coding will be described with reference to FIG.

FIG. 2A is a multi-viewpoint image group (sometimes simply referred to as a multi-viewpoint image), and the images of the two viewpoints in the central portion of the image group are enlarged and shown at the top. The multi-viewpoint image group is, for example, an image taken by a camera array, and the upper two images correspond to an image of an object taken by an adjacent camera. Each image constituting the multi-viewpoint image group corresponds to an image of one viewpoint, and each viewpoint image causes parallax with respect to an object. It should be noted that each viewpoint image constituting the multi-viewpoint image group may include not only the image actually taken but also the image created by viewpoint interpolation or the like. The entire multi-viewpoint image group (22 × 22 = 484 viewpoints) in FIG. 2 (A) is an input image, and the image with a circle is used for conversion to a coding element image group. It is a multi-viewpoint image (in this example, 8 × 8 = 64 viewpoints). The image of the viewpoint without the circle is thinned out by the viewpoint thinning unit 11 in order to reduce the amount of data, and is not used for the subsequent processing.

FIG. 2B is a group of element images for coding, and a plurality of element images in the central portion are enlarged and shown at the upper part. In general, the element image group can be created by converting from the multi-viewpoint image group. That is, one element image is generated by extracting one pixel at the same coordinate position from each viewpoint image constituting the multi-viewpoint image group and accumulating the multi-viewpoint image group while maintaining the overall arrangement. .. For example, an 8 × 8 pixel element image (encoding element image) is generated from one pixel of each viewpoint image of the multi-viewpoint image group taken by the 8 × 8 cameras in FIG. 2 (A). .. By generating other element images in the same manner, the multi-viewpoint image group can be converted into a coding element image group in which the element images of 8 × 8 pixels are set by the number of pixels of the viewpoint image.

The multi-viewpoint image element image conversion unit 12 converts the thinned-out multi-viewpoint image group (A) into the coding element image group (B) by such processing, and outputs the image to the coding unit 13. ..

The coding unit 13 encodes the input coding element image group. The coding process performs compression / coding by a coding tool having an intra-block copy function (for example, an extended standard of H.265 / HEVC (High Efficiency Video Coding)) among the conventional coding tools. By using the intra-block copy function that predicts the block to be coded from the adjacent block, it is possible to perform prediction processing using the high correlation between the element images of the element image group for coding, and the amount of coded data is greatly increased. Can be reduced to. In order to perform highly efficient data compression by intra-block copying, it is desirable that the pixel size of the element image (coding element image) of the coding element image group be equal to the coding block unit.

The image coding data generated by the coding unit 13 is output as the output of the coding device 10.

[Decoding device]
The decoding device 20 includes a decoding unit 21, an element image multi-viewpoint image conversion unit 22, a depth estimation unit 23, a viewpoint interpolation processing unit 24, and a multi-viewpoint image element image conversion unit 25.

The image coded data encoded by the coding device 10 is input to the decoding unit 21. The decoding unit 21 decodes the input image-encoded data by a decoding method corresponding to the coding. The decoded image data is an element image group (for coding). The decoded element image group is output to the element image multi-viewpoint image conversion unit 22.

The element image multi-viewpoint image conversion unit 22 converts the input element image group into a multi-viewpoint image group. Since the input element image group is substantially the coding element image group (FIG. 2B) that is the object of coding, the conversion from the multi-viewpoint image group to the element image group described above is performed. By performing the completely reverse conversion, the element image group is converted into the 8 × 8 multi-viewpoint images (thinned multi-viewpoint image group) shown in FIG. 2 (A). The element image multi-viewpoint image conversion unit 22 outputs the converted multi-viewpoint image group to the depth estimation unit 23 and the viewpoint interpolation processing unit 24.

The depth estimation unit 23 estimates the depth from the input multi-viewpoint image group and generates a depth map of the multi-viewpoint image group. The depth information of the image is reflected in the depth map. The generated depth map is output to the viewpoint interpolation processing unit 24.

The viewpoint interpolation processing unit 24 generates a viewpoint image between the decoded viewpoints (cameras) by using the decoded multi-viewpoint image group and the generated depth map. That is, from the decoded 8 × 8 multi-viewpoint images (viewpoints marked with a circle in FIG. 2A), the viewpoint images between the viewpoints are predicted and interpolated by the viewpoint interpolation processing, and 22 before the thinning process is performed. × 22 = A multi-viewpoint image group of 484 viewpoints (entire of FIG. 2A) is reproduced. The reproduced multi-viewpoint image group is output to the multi-viewpoint image element image conversion unit 25.

The multi-viewpoint image element image conversion unit 25 converts the input multi-viewpoint image group into an element image group. FIG. 3 shows an example of the input multi-viewpoint image group (FIG. 3 (A)) and the element image group (FIG. 3 (B)). The multi-viewpoint image group input to the multi-viewpoint image element image conversion unit 25 is a 22 × 22 = 484-viewpoint multi-viewpoint image group consisting of a decoded multi-viewpoint image and an interpolated viewpoint image between the generated cameras. Yes, it corresponds to a restored input image input to the encoding device 10. By converting this into an element image group, an element image group of an element image (22 × 22 pixels) having a large pixel size is generated, and this is used as an output image of the decoding device 20. With this output image, an integral 3D image can be displayed.

In the element image group of the output image, the number of pixels of each element image can be increased as compared with the element image group of the transmitted image coding data, and an integral 3D image having a wider depth can be reproduced.

In the present embodiment, the element image group is used as the output image on the premise that the integral 3D image is displayed based on the output image. For example, in order to display the multi-view image, the multi-view image element is used. The multi-viewpoint image after the viewpoint insertion may be used as the output image of the decoding device without providing the image conversion unit 25.

According to the present embodiment, the amount of coded data can be significantly reduced by thinning out the viewpoints from the multi-viewpoint image group and the coding process using the high correlation between the element images of the coding element image group. it can. Further, in the decoding device, highly accurate viewpoint interpolation can be performed by using the depth map.

(Second embodiment)
FIG. 4 shows an example of a block diagram of the coding device and the decoding device according to the second embodiment of the present invention. The second embodiment is an example of encoding a 2D image as one data format in addition to a multi-viewpoint image group, and is applied to a broadcasting system that transmits information of both an integral 3D image and a 2D broadcast image. It is a coding / decoding system capable of. The coding device 10-1 and the decoding device 20-1 may be connected by an arbitrary transmission line capable of information communication, and in this case, both function as a transmitting device 10-1 and a receiving device 20-1. To do. As a transmission / reception method at this time, a broadcasting system, radio wave communication, a wired / wireless network, or the like can be used. In addition, both may be used as independent devices, and data may be exchanged using a recording medium or the like.

Hereinafter, each of the coding device 10-1 and the decoding device 20-1 will be described in detail. The parts common to FIG. 1 will be simplified.

[Coordinator]
The coding device 10-1 includes a viewpoint thinning unit 11, a multi-view image element image conversion unit 14, and a coding unit 13, and a multi-view image group and a 2D image are input.

One of the input images is, for example, a multi-viewpoint image group acquired by a multi-viewpoint camera, and is input to the viewpoint thinning unit 11. This input image may be the same as that of the first embodiment, and is, for example, a multi-viewpoint image group for an integral 3D image in which 22 × 22 (= 484) cameras are arranged vertically and horizontally. Each of the images from one viewpoint is a color texture image.

The other of the input images is a 2D image, for example, a 2D broadcast image. The 2D image is input to the multi-viewpoint image element image conversion unit 14. In the present embodiment, it is assumed that the 2D image is an image having three times the number of pixels in the vertical and horizontal directions as compared with the image of one viewpoint in the multi-viewpoint image group.

The viewpoint thinning unit 11 performs a viewpoint thinning process for thinning the viewpoints at equal intervals with respect to the input multi-view image. Further, in order to embed a 2D image, the viewpoint of a predetermined area is thinned out in advance. For example, the 22 × 22 viewpoint is thinned out to reduce the image to an 8 × 8 viewpoint image, and the 3 × 3 viewpoint in a predetermined area (area in which the 2D image is embedded) is excluded from the 8 × 8 viewpoint image. The thinned out multi-viewpoint image group is output to the multi-viewpoint image element image conversion unit 14.

The multi-viewpoint image element image conversion unit 14 combines the thinned-out multi-viewpoint image group and the input 2D image and converts them into a coding element image group. The conversion to the coding element image group in this embodiment will be described with reference to FIG.

In FIG. 5A, the upper image is a 2D image input to the coding device, and is, for example, a 2D broadcast image. In the present embodiment, the 2D image is an image having three times the number of pixels in the vertical and horizontal directions as compared with the image of one viewpoint in the multi-viewpoint image group. However, in FIG. 5, the screen size of the 2D image is exaggerated.

The entire lower image of FIG. 5 (A) is a multi-viewpoint image group (sometimes simply referred to as a multi-viewpoint image) input to the encoding device, and here, the multi-viewpoint image shown in FIG. 2 (A). It is the same as the viewpoint image group. For example, it is an image taken by a camera array of 22 × 22 = 484 viewpoints, each image constituting a multi-viewpoint image group corresponds to an image of one viewpoint, and each viewpoint image has parallax with respect to an object. Occurs. Of the multi-viewpoint image group of FIG. 5A, the image with a circle is a multi-viewpoint image used for conversion to a coding element image group after being thinned out by the viewpoint thinning unit 11. is there. In this example, after 8 × 8 = 64 viewpoints are extracted at equal intervals, the 3 × 3 = 9 viewpoints in the central part are further removed, and 55 viewpoints are used for conversion to the element image group for coding. ..

FIG. 5B is a group of element images for coding, and a plurality of element images in the central portion are enlarged and shown in the upper part. One pixel at the same coordinate position from each of the 55 viewpoint images constituting the multi-viewpoint image group is extracted and accumulated while maintaining the overall arrangement of the multi-viewpoint image group. Further, in the present embodiment, the screen resolution of the 2D image is three times as large as the screen resolution of the multi-view image, so that the 3 × 3 of the 2D image at the position corresponding to one pixel of the multi-view image. One element image is generated by extracting pixels as one pixel block and arranging them in a region of 3 × 3 pixels in the center of a coding element image. As a result, as shown in FIG. 5C, an element image of 8 × 8 (= 64) pixels is obtained by combining each 1 pixel of 55 multi-viewpoint image groups and 3 × 3 pixels of a 2D image. (Element image for coding) is generated. By generating other element images in the same manner, a multi-viewpoint image group and a 2D image are assembled, and an 8 × 8 pixel element image is a set of the number of pixels of the viewpoint image. It can be converted into an element image group for conversion).

The multi-viewpoint image element image conversion unit 14 converts the thinned-out multi-viewpoint image group and the 2D image into the coding element image group (B) by such processing, and outputs the image to the coding unit 13. .. The coding element image group (B) is an example. For example, when the 2D image is an image having twice the number of pixels in the vertical and horizontal directions as the image of one viewpoint in the multi-viewpoint image group. The 2D image to be embedded in the element image is a pixel block of 2 × 2 pixels. When the 2D image is an image having the same number of pixels as the image of one viewpoint in the multi-view image group, the 2D image is also embedded in the element image as one pixel. The position where the 2D image is embedded is not limited to the center of the element image, and may be embedded at any position in the element image.

The coding unit 13 encodes the input coding element image group. The coding unit 13 is the same as the coding unit 13 of FIG. 1, and the coding process performs compression / coding by a coding tool having an intra-block copy function among the conventional coding tools. By using the intra-block copy function for the coding element image group in which the 2D image is embedded, the amount of coded data can be significantly reduced. It is desirable that the pixel size of the element image for coding be equal to the block unit of coding.

The image coding data generated by the coding unit 13 is output as the output of the coding device 10-1.

[Decoding device]
The decoding device 20-1 includes a decoding unit 21, an element image multi-viewpoint image conversion unit 26, a depth estimation unit 23, a viewpoint interpolation processing unit 24, and a multi-viewpoint image element image conversion unit 25.

The image coded data encoded by the coding device 10-1 is input to the decoding unit 21. The decoding unit 21 decodes the input image-encoded data by a decoding method corresponding to the coding. The decoded image data is an element image group for coding in which a 2D image is embedded. The decoded coding element image group is output to the element image multi-viewpoint image conversion unit 26.

The element image multi-viewpoint image conversion unit 26 extracts the pixels of the 2D image from the input coding element image group and converts the coding element image group into the multi-viewpoint image group. That is, since each element image of the decoded coding element image group has the pixel arrangement shown in FIG. 5C, the central 2D image (3 × 3 pixels) is extracted from each element image. , This is accumulated based on the position of each element image in the element image group for coding, and the original 2D image is restored. Further, for the other pixels of each element image (pixels based on the multi-viewpoint image), the element image group for coding can be obtained by performing the conversion completely opposite to the conversion from the multi-viewpoint image group to the element image group described above. It is converted into the 55 multi-viewpoint image group shown in FIG. 5 (A). The element image multi-viewpoint image conversion unit 26 outputs the restored 2D image as an output image of the decoding device 20-1, and outputs the converted multi-viewpoint image group to the depth estimation unit 23 and the viewpoint interpolation processing unit 24.

The depth estimation unit 23 estimates the depth from the input multi-viewpoint image group, creates a depth map of the multi-viewpoint image group, and outputs the created depth map to the viewpoint interpolation processing unit 24.

The viewpoint interpolation processing unit 24 generates a thinned-out viewpoint by viewpoint interpolation by using the decoded multi-viewpoint image group and the depth map of the multi-viewpoint image group. That is, from the 55 multi-viewpoint images after decoding (the viewpoints marked with circles in FIG. 5 (A)), a multi-viewpoint image group of 22 × 22 = 484 viewpoints before the thinning process is performed by the viewpoint interpolation processing (FIG. 5 (A) as a whole) is reproduced. The reproduced multi-viewpoint image group is output to the multi-viewpoint image element image conversion unit 25.

The multi-viewpoint image element image conversion unit 25 converts the input multi-viewpoint image group into an element image group. By converting the multi-viewpoint image group interpolated into the viewpoint into the element image group, the element image group of the element image (22 × 22 pixels, FIG. 3B) having a large pixel size is generated, and the element image group is generated by the decoding device 20. The output image is -1. An integral 3D image is displayed by this output image.

In the present embodiment, the element image group is the output image of the decoding device 20-1 on the premise that the integral 3D image is displayed. For example, when displaying the multi-viewpoint image based on the output image, the element image group is used as the output image. May omit the multi-viewpoint image element image conversion unit 25 and use the multi-viewpoint image as an output image.

According to this embodiment, the integral 3D image and the conventional 2D broadcast image can be simultaneously compressed and encoded and broadcast. As a result, in the case of a display monitor that supports only conventional 2D broadcast images, display processing is performed using 2D images, and in the case of a display monitor that supports integral 3D images, element images are used. Display processing can be performed.

According to the present embodiment, the coding apparatus can simultaneously encode and transmit a multi-viewpoint image group and a 2D image, and is highly efficient with respect to an image for an integral 3D image while applying a conventional video coding method. Data can be compressed. Further, the decoding device can output the element image group and the 2D image, and can select the image according to the application.

In the above-described embodiment, the configuration and operation of the coding devices 10 and 10-1 have been described, but the present invention is not limited to this, and may be configured as a coding method for coding a multi-viewpoint image. That is, it is configured as a coding method for generating image-coded data from a multi-viewpoint image or a coding method for generating image-coded data from a multi-viewpoint image and a 2D image according to the data flow of FIG. 1 or FIG. You may. Further, although the configurations and operations of the decoding devices 20 and 20-1 have been described, the present invention is not limited to this, and may be configured as a decoding method for decoding image-encoded data. That is, according to the data flow of FIG. 1 or 4, the decoding method for generating the output image of the element image group from the image coded data, or the output image of the element image group and the 2D image is generated from the image coded data. It may be configured as a decoding method.

A computer can be preferably used to function as the above-mentioned coding devices 10, 10-1 or decoding devices 20, 20-1, and such a computer can be used as the coding devices 10, 10-1 or decoding. It can be realized by storing a program describing the processing contents for realizing each function of the devices 20 and 20-1 in the storage unit of the computer, and reading and executing this program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

Although the above embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and modifications can be made without departing from the scope of claims. For example, it is possible to combine the plurality of constituent blocks described in the embodiment into one, or to divide one constituent block into one.

10 Coding device 11 Viewpoint thinning unit 12 Multi-viewpoint image element image conversion unit 13 Coding unit 14 Multi-viewpoint image element image conversion unit 20 Decoding device 21 Decoding unit 22 Element image Multi-viewpoint image conversion unit 23 Depth estimation unit 24 Viewpoint insertion Processing unit 25 Multi-viewpoint image Element image conversion unit 26 Element image Multi-viewpoint image conversion unit

Claims (9)

A viewpoint thinning unit that performs viewpoint thinning processing on the input multi-view image group,
A multi-viewpoint image element image conversion unit that converts the multi-viewpoint image group thinned out from the viewpoint into an element image group for coding,
A coding device including a coding unit that encodes the coding element image group. In the coding device according to claim 1, the coding process uses a coding tool having an intra-block copy function. The coding device according to claim 1 or 2, wherein the pixel size of the element image of the coding element image group is equal to the coding block unit. In the coding apparatus according to any one of claims 1 to 3, a 2D (two-dimensional) image is further input, and the multi-viewpoint image element image conversion unit performs each element image of the coding element image group. An encoding device that embeds the corresponding pixel or pixel block of the 2D image in the image. A decoding unit that decodes the input image coding data and creates an element image group for coding,
An element image multi-viewpoint image conversion unit that converts the coding element image group into a multi-viewpoint image group,
A depth estimation unit that estimates the depth of each multi-view image based on the multi-view image group and generates a depth map,
A decoding device including a viewpoint insertion processing unit that performs viewpoint interpolation between viewpoints of the multi-view image group based on the multi-view image group and the generated depth map. The decoding device according to claim 5, further comprising a multi-viewpoint image element image conversion unit that converts a multi-viewpoint image group interpolated into a viewpoint into an element image group, and outputs the element image group. In the decoding device according to claim 5 or 6, a 2D (two-dimensional) image is embedded in the coding element image group, and the element image multi-viewpoint image conversion unit is the coding element image group. A decoding device that extracts pixels or pixel blocks of the 2D image from each element image of the above, integrates them, reproduces the 2D image, and outputs the image. A program that causes a computer to function as the coding device according to any one of claims 1 to 4. A program that causes a computer to function as the decoding device according to any one of claims 5 to 7.

Images