JP7417388B2 - Encoding device, decoding device, and program - Google Patents

Description

The present invention relates to an encoding device, a decoding device, and a program, and in particular to an encoding device, a decoding device, and a program for multi-view images necessary for displaying integral 3D (three-dimensional) video and free-view video. Regarding.

A light field camera in which a lens array is arranged in front of a sensor of an image pickup device has been commercialized as a camera capable of photographing a group of elemental images that display an integral 3D image. However, light field cameras are generally intended for a refocusing function after shooting. Therefore, when displaying integral 3D images using images taken with a light field camera, the diameter of the main lens that makes up the light field camera is small compared to the distance to the subject, so motion parallax is small. , it is not possible to sufficiently reproduce the depth of 3D images. This problem can theoretically be solved by increasing the diameter of the main lens or shortening the distance between the camera and the subject, but these measures are not practical.

Therefore, it has been considered to capture multi-view images using a camera array in which ordinary cameras are arranged in a two-dimensional array horizontally and vertically. In this case, the element image group is generated by using viewpoint interpolation processing from multiple images taken by the camera array to generate viewpoint images between the cameras, and then from the images taken by the camera array and the viewpoint interpolation images. A process of converting into a group of elemental images is performed (Patent Document 1). Here, it is known that the inter-camera distance of the camera array can be designed based on the distance from the camera to the subject, the distance where viewpoint interpolation is practically possible, and the viewing angle that can be reproduced by the display device. Furthermore, in viewpoint interpolation processing, a highly accurate interpolated image is generated by using a depth map (depth image) that relatively expresses the distance from the camera to the subject. The depth map is generated by estimating depth using image processing technology or optically measuring distance using infrared rays. Increasing the accuracy of this depth map generation also improves the accuracy of viewpoint interpolation.

Regarding the display of an integral 3D image, the depth at which a 3D image can be reproduced is related to the parallax between adjacent multi-view images, the focal length of the lens array, and the number of pixels of the element images. Among these, it is known that increasing the number of pixels of elemental images is effective in increasing the depth at which three-dimensional images can be reproduced. In this case, the number of pixels in the elemental image is equal to the number of viewpoints in the multi-view image, so in order to generate a three-dimensional video with depth, a large number of viewpoints in the multi-view image to be encoded is required. The amount of information needed to display three-dimensional images is enormous.

In transmitting and recording integral 3D video, a huge amount of information is encoded to display the 3D video. In encoding, methods are known in which multi-view video encoding is performed after converting a group of elemental images into a group of multi-view images, and a method in which the converted multi-view video is thinned out at the time of encoding and viewpoint interpolation is performed at the time of decoding. In this method, the number of images to be encoded is reduced by thinning out multi-view images, thereby reducing the amount of information required to display a three-dimensional image.

FIG. 9 shows an example of a conventional encoding target image. FIG. 9 shows a multi-view video and one frame of a depth map corresponding to the multi-view video, and a collection of N frames constitutes a video. In each frame, for example, a multi-view image of 13 x 13 = 169 viewpoints is thinned out at equal intervals to create a multi-view image of 5 x 5 = 25 viewpoints, and similarly, a depth map of 13 x 13 = 169 viewpoints is thinned out at equal intervals. The depth map is thinned out and has 5×5=25 viewpoints. Images represented by broken lines are multi-view images that are not encoded. In the encoding target image, multi-view images and depth maps of the same viewpoint are arranged at equal intervals.

In encoding this encoding target image, the high correlation between adjacent viewpoint images (an image of one viewpoint among multi-view images is referred to as a "viewpoint image") is used to A method of encoding by determining the difference between a viewpoint image and a viewpoint image to be encoded, and a method of encoding using the high correlation between a viewpoint image and a corresponding depth map, etc. The multi-view image and the depth map are integrally encoded as one frame, resulting in high compression rate encoding.

Furthermore, on the decoding side, after decoding the encoding target image, viewpoint interpolation is performed for the thinned out (missing) multi-view images. In viewpoint interpolation, a depth map is first interpolated, and for example, a depth map of 13×13=169 viewpoints is reproduced from a depth map of 5×5=25 viewpoints. Next, interpolation of viewpoint images between the decoded viewpoints is performed based on the depth maps of all viewpoints and the decoded viewpoint images.

Japanese Patent Application Publication No. 2016-158213

In order to display an integral 3D video with greater depth, a group of elemental images with a large number of pixels is required, and therefore a large number of viewpoints in a multi-view image is required. In order to compress the data amount of a multi-view image with a large number of viewpoints, a method can be considered to increase the thinning of the viewpoints, widen the interval between the viewpoints to be encoded, and reduce the number of images to be encoded.

However, while the method of reducing the number of images to be encoded by thinning out many viewpoints can reduce the amount of information needed to display 3D images during transmission and recording, it also causes deterioration in image quality due to viewpoint interpolation after decoding. do. This is because when generating the depth map required for viewpoint interpolation, the number of images to be encoded is reduced, so the distance between adjacent images increases and the accuracy of the generated depth map decreases. Another cause is that the accuracy of the in-painting process for filling in the occlusion area decreases because the parallax increases in the prediction of viewpoint interpolation. Image quality deterioration due to viewpoint interpolation includes image quality deterioration due to viewpoint interpolation of depth maps and image quality deterioration due to viewpoint interpolation of multi-view images. Furthermore, in the encoding process, since the parallax between the multi-view images to be encoded increases, the accuracy of viewpoint compensation prediction decreases, and the encoding efficiency deteriorates.

Therefore, an object of the present invention, which was made in view of the above-mentioned problems, is to efficiently reduce the amount of transmitted data of multi-view videos and depth maps in encoding, and to reduce the image quality deterioration of viewpoint interpolated images in decoding. An object of the present invention is to provide an encoding device, a decoding device, and a program that can suppress the problem.

In order to solve the above problems, an encoding device according to the present invention is an encoding device that encodes a multi-view video and a depth map corresponding to the multi-view video, and the encoding device encodes a multi-view video and a depth map corresponding to the multi-view video in each frame. A viewpoint thinning unit that thins out the viewpoints of the map, a frame thinning unit that thins out the frames of the multi-view video and the depth map, and an encoding unit that encodes the multi-view video and the depth map that have been thinned out by the viewpoint thinning and frames. In the encoding device, the frame thinning unit thins out either the viewpoint image or the depth map at each viewpoint of each frame, and switches the thinning target for each frame.

Further, in the encoding device, the frame thinning unit thins out one of the viewpoint image and the depth map of one viewpoint in each frame, and the other of the viewpoint image and the depth map in at least one viewpoint adjacent to the one viewpoint. It is desirable to thin out.

Further, in the encoding device, it is preferable that the frame thinning unit thins out the viewpoint image or the depth map in each viewpoint thinned frame so as to form a checkered pattern.

Further, in the encoding device, it is preferable that the frame thinning unit thins out the viewpoint images and the depth map in each frame thinned out so that the viewpoint images and the depth map are arranged alternately in columns or rows.

In order to solve the above problems, a decoding device according to the present invention includes a decoding unit that decodes a multi-view video and a depth map from encoded video data, and a decoding unit that decodes a multi-view video and a depth map from video encoded data, and and a viewpoint interpolation unit that interpolates the thinned-out viewpoint image and the depth map in each frame, and the viewpoint interpolation unit interpolates the viewpoint image and the depth map decoded from the video encoded data. The reliability of the map is set to a higher value than the reliability of the interpolated viewpoint image and the depth map, and the viewpoint interpolation unit predicts the viewpoint image based on the reliability weight of the reference image. It is characterized by doing the following .

In order to solve the above problems, a program according to the present invention is characterized by causing a computer to function as the above encoding device.

In order to solve the above problems, a program according to the present invention is characterized by causing a computer to function as the above decoding device.

According to the encoding device, decoding device, and program of the present invention, it is possible to efficiently reduce the amount of transmission data of multi-view videos and depth maps during encoding, and suppress image quality deterioration of viewpoint interpolated images during decoding. I can do it.

It is an example of a block diagram of an encoding device and a decoding device of the present invention. FIG. 3 is a diagram illustrating an example of an image to be encoded in an even-numbered frame. FIG. 3 is a diagram illustrating an example of an image to be encoded in an odd-numbered frame. FIG. 7 is a diagram illustrating another example of an image to be encoded in an even-numbered frame. FIG. 7 is a diagram illustrating another example of an image to be encoded in an odd-numbered frame. FIG. 7 is a diagram showing still another example of an image to be encoded in an even frame. FIG. 7 is a diagram illustrating yet another example of an image to be encoded in an odd-numbered frame. FIG. 3 is a diagram illustrating viewpoint interpolation processing of the present invention. FIG. 2 is a diagram showing an example of a conventional encoding target image.

Embodiments of the present invention will be described below.

FIG. 1 shows an example of a block diagram of an encoding device and a decoding device of the present invention. Encoding device 10 and decoding device 20 constitute an encoding/decoding system as a whole. Encoding device 10 and decoding device 20 may be connected by any transmission path that allows information communication, and in this case, both function as transmitting device 10 and receiving device 20. As a transmission/reception method at this time, a broadcasting system, radio wave communication, wired/wireless network, etc. can be used. Alternatively, both may be independent devices, and data may be sent and received from the encoding device 10 to the decoding device 20 using a recording medium or the like.

Each of the encoding device 10 and the decoding device 20 will be described in detail below.

[Encoding device]
The encoding device 10 includes a viewpoint thinning section 11, a frame thinning section 12, and an encoding section 13.

The input image is, for example, a multi-view video obtained by a multi-view camera in which cameras (for example, CMOS sensors) are arranged in 13×13 (=169) rows and columns, and a depth map corresponding to the multi-view video. Each one-viewpoint image (viewpoint image) of the multi-view video is a color texture image, and the depth map is a multi-view depth image having the same viewpoint and frame as the multi-view video. The input image is input to the viewpoint thinning unit 11 .

Note that in this embodiment, the depth map is generated externally and inputted to the encoding device 10, but only multi-view video is input to the encoding device 10 as an input video, and the depth map is generated externally and inputted to the encoding device 10. A depth map can also be generated within the encoding device 10 by performing depth estimation processing or the like.

The viewpoint thinning unit 11 performs a viewpoint thinning process of thinning out the viewpoints at equal intervals in each frame on the input multi-view video and the depth map corresponding to the multi-view video. For example, a multi-view image with 13×13=169 viewpoints is thinned out at equal intervals to reduce it to a multi-view image with 5×5=25 viewpoints. Similarly, for the depth map, the depth map with 13×13=169 viewpoints is thinned out to create a depth map with 5×5=25 viewpoints. The multi-view image and depth map obtained as a result of viewpoint thinning are the same as the conventional encoding target image (encoding target video) in FIG. They are arranged at equal intervals. The multi-view video and depth map with the viewpoints thinned out are output to the frame thinning unit 12.

The frame thinning unit 12 thins out the frames of the input (viewpoint thinned out) multi-view video and depth map. Here, an example of frame thinning processing will be described using FIGS. 2 and 3.

Figure 2 shows the even-numbered (2n: n is an integer) frame of the multi-view video and depth map with frame thinning, and Figure 3 shows the odd-numbered (2n+1) frame of the frame-thinned multi-view video and depth map. Showing eyes. That is, FIG. 2 shows an example of an image to be encoded in an even frame, and FIG. 3 shows an example of an image to be encoded in an odd frame. In each frame, one of the multi-view image and the depth map is thinned out (frame thinning) among 5×5=25 thinned-out viewpoints. As is clear from comparing FIG. 2 and FIG. 3, in a certain viewpoint, when the depth map is thinned out in even frames, the viewpoint image is thinned out in odd frames. For viewpoints whose viewpoint images are thinned out in even-numbered frames, depth maps are thinned out in odd-numbered frames. That is, the thinning target of the viewpoint image and the depth map is alternately switched for each frame.

Furthermore, in each frame where the viewpoints have been thinned out, when the depth map is thinned out at a certain viewpoint, it is desirable to thin out the viewpoint images at at least one viewpoint adjacent to that viewpoint (after viewpoint thinning). That is, when one of the viewpoint image and the depth map is thinned out at one viewpoint, it is desirable to thin out the other of the viewpoint image and the depth map at at least one viewpoint adjacent to the one viewpoint. As a result, the viewpoint images and depth maps are respectively encoded at adjacent viewpoints. Then, when viewpoint interpolation is performed on the decoding side, at least one of the adjacent viewpoint images becomes a highly accurate decoded image.

In the example in Figure 2, when the depth map is thinned out at a certain viewpoint and the viewpoint image is used as the encoding target image, the adjacent viewpoints on the upper, lower, left, and right sides (after viewpoint thinning) are thinned out and the depth map is used as the encoding target image. Make it an image. Even in the odd-numbered frames in FIG. 3, the positions of the viewpoint image and the depth map are opposite, but the viewpoint image and the depth map alternately serve as encoding target images for a certain viewpoint and the adjacent viewpoints on the upper, lower, left, and right sides. In this way, in FIGS. 2 and 3, the frames are thinned out so that the perspective images (or depth maps) form a checkered pattern in each frame that has been thinned out.

Another example of frame thinning processing will be described using FIGS. 4 and 5.

FIG. 4 shows the even-numbered (2n: n is an integer) frame of the multi-view video and depth map with frame thinning, and FIG. 5 shows the odd-numbered (2n+1) frame of the multi-view video with frame thinning and the depth map. Showing eyes. That is, FIG. 4 shows another example of an image to be encoded in an even frame, and FIG. 5 shows another example of an image to be encoded in an odd frame. In each frame, one of the multi-view image and the depth map is thinned out (frame thinning) among 5×5=25 thinned-out viewpoints. As is clear from comparing FIG. 4 and FIG. 5, in a certain viewpoint, when the depth map is thinned out in even frames, the viewpoint image is thinned out in odd frames, and the viewpoint image is thinned out in even frames. In odd frames, the depth map is thinned out.

In the example shown in FIG. 4, when the depth map is thinned out at a certain viewpoint and the viewpoint image is used as the encoding target image, the adjacent viewpoints on the left and right (after viewpoint thinning) are thinned out and the depth map is used as the encoding target image. shall be. Even in the odd-numbered frames in FIG. 5, the positions of the viewpoint image and the depth map are reversed, but the viewpoint image and the depth map are alternately encoding target images in a certain viewpoint and the adjacent left and right viewpoints. In this way, in FIGS. 4 and 5, it is desirable that the viewpoint images and depth maps are thinned out so that they are alternately arranged in columns in each frame where the viewpoints have been thinned out.

Still another example of frame thinning processing will be described with reference to FIGS. 6 and 7.

FIG. 6 shows the even-numbered (2n: n is an integer) frame of the frame-thinned multi-view video and depth map, and FIG. 7 shows the odd-numbered (2n+1) frame of the frame-thinned multi-view video and depth map. Showing eyes. That is, FIG. 6 shows yet another example of the image to be encoded in the even-numbered frame, and FIG. 7 shows still another example of the image to be encoded in the odd-numbered frame. In each frame, one of the multi-view image and the depth map is thinned out (frame thinning) in 5×5=25 viewpoints, and as is clear from comparing FIG. 6 and FIG. The image and depth map thinning targets are alternately switched for each frame.

In the example of FIG. 6, when the depth map is thinned out at a certain viewpoint and the viewpoint image is used as the encoding target image, the adjacent viewpoints above and below (after viewpoint thinning) are thinned out and the depth map is used as the encoding target image. shall be. Even in the odd-numbered frames in FIG. 7, the positions of the viewpoint image and the depth map are opposite, but the viewpoint image and the depth map are alternately encoding target images in a certain viewpoint and the adjacent upper and lower viewpoints. In this way, in FIGS. 6 and 7, it is desirable that the viewpoint images and depth maps are thinned out so that they are alternately arranged in rows in each frame where the viewpoints have been thinned out.

Returning to FIG. 1, the frame-thinned multi-view video and depth map (multi-view image and depth map encoding target image) are output to the encoding unit 13.

The encoding unit 13 encodes the frame-thinned multi-view video and the depth map (the input encoding target image). In the encoding process, compression and encoding are performed using a conventionally used encoding tool (for example, an extended standard of H.265/HEVC (High Efficiency Video Coding), etc.). Highly efficient data compression can be performed by prediction processing that utilizes the high correlation between the multi-view video to be encoded and the depth map.

The encoded video data generated by the encoder 13 is output as the output of the encoder 10.

According to the encoding device of this embodiment, although the distance between viewpoints of the encoding target image (FIGS. 2 to 7) is the same as that of the conventional example (FIG. 9), the viewpoint image of each viewpoint and the depth map frame are The data are alternately thinned out, and the amount of information to be encoded can be reduced by half.

[Decoding device]
The decoding device 20 includes a decoding section 21, a frame interpolation section 22, a viewpoint interpolation section 23, and a multi-view image element image conversion section 24.

Video encoded data encoded by the encoding device 10 is input to the decoding unit 21 . The decoding unit 21 decodes the input video encoded data using a decoding method corresponding to encoding. Through the decoding process, the frame-thinned multi-view video and the depth map (encoding target image) are decoded. The decoded video (image) data is output to the frame interpolation section 22.

The frame interpolation unit 22 performs frame interpolation between the frame-thinned multi-view video and the depth map. For example, at a viewpoint where the depth map is thinned out in 2n frames, the depth map of 2n frames is predicted by interpolation processing using the depth map of the same viewpoint of 2n-1 frame and the depth map of the same viewpoint of 2n+1 frame. and interpolate the depth map frame. Note that intra prediction may be used using adjacent depth maps within 2n frames. Frame interpolation is similarly performed on the thinned depth map of odd number (2n+1) frames.

Regarding frame interpolation of multi-view video, for example, for a viewpoint where the viewpoint images are thinned out in 2n frames, the viewpoint images of the same viewpoint of 2n-1 frames and the viewpoint images of the same viewpoint of 2n+1 frames are used. In addition to the interpolation process, it is desirable to use the depth map of the 2n-frame viewpoint to predict the 2n-frame viewpoint image and interpolate the frame of the viewpoint image. The depth map of the viewpoint is a highly accurate depth map obtained by decoding the encoded data, and it is possible to improve the accuracy of the viewpoint image for frame interpolation. Frame interpolation is similarly performed on the thinned-out viewpoint images of odd number (2n+1) frames.

In this way, frame interpolation is performed between the frame-thinned multi-view video and the depth map at each viewpoint of each frame. As a result, the information amount of the multi-view video and the depth map after frame interpolation, and the positional relationship between the viewpoints are the same as in FIG. 9 . The frame interpolated multi-view video and depth map are output to the viewpoint interpolation unit 23.

The viewpoint interpolation unit 23 uses the frame-interpolated multi-view video and the depth map to generate an uncoded multi-view image between viewpoints by viewpoint interpolation. That is, from the 5 x 5 multi-view images after frame interpolation and the depth map (view thinned images: see Figure 9), the viewpoint images between viewpoints are predicted and interpolated by viewpoint interpolation processing, and the viewpoint A multi-view image of 13×13=169 viewpoints (the entirety of FIG. 9) before thinning is reproduced.

The viewpoint interpolation process in the viewpoint interpolation unit 23 will be explained with reference to FIG. In FIG. 8, each viewpoint is represented by two-dimensional coordinates based on the upper left viewpoint. For example, a viewpoint image at coordinates (1,1) and a viewpoint image at coordinates (4,4) are viewpoint images decoded from video encoded data, and a viewpoint image at coordinates (1,4) and a viewpoint image at coordinates (4,4) are decoded from video encoded data. It is assumed that the viewpoint image in 1) is a viewpoint image generated by frame interpolation. Also, the depth map at coordinates (1, 4) and the depth map at coordinates (4, 1) are depth maps decoded from video encoded data, and the depth map at coordinates (1, 1) and the depth map at coordinates (4, 1) are depth maps decoded from video encoded data. It is assumed that the depth map 4) is a depth map generated by frame interpolation.

In this embodiment, reliability weights (coefficients) of a multi-view image that is a reference image and a depth map are set. This reliability setting reduces the reliability of multi-view images and depth maps generated by frame interpolation, and increases the reliability of multi-view images and depth maps decoded from transmitted and recorded video encoded data. Set to value.

In viewpoint interpolation, viewpoint interpolation of a depth map is first performed as in the conventional method. That is, from the depth maps of 5×5=25 viewpoints, all depth maps between viewpoints are generated and interpolated. In the example of FIG. 8, when generating a depth map for a certain viewpoint (for example, coordinates (3, 3)), the decoded or frame interpolated depth map (coordinates (1, 1), (1, 4), (4, 1), (4, 4)) as a reference image, interpolate the image based on the distance from the reference image (for example, using the reciprocal of the distance as the reliability), and Generate a depth map.

In this embodiment, at this time, the weight of the reliability of the depth map, which is the reference image, is added as a calculation element. In other words, the reliability based on the depth map (coordinates (1, 1), (4, 4)) generated by frame interpolation is corrected to be smaller than the reliability based on the distance from the reference image, and the The reliability based on the depth map (coordinates (1, 4), (4, 1)) decoded from the encoded data is corrected to a high value. For example, the reliability of an interpolated depth map is corrected with a weighting factor of 0.5, and the reliability of a decoded depth map is corrected with a weighting factor of 1.0. Note that the coefficient can be adjusted. Based on the corrected reliability, interpolation processing is performed based on the reference image to generate and interpolate a depth map between viewpoints. Similar processing is performed for depth maps of other viewpoints.

Next, after viewpoint interpolation of all depth maps, viewpoint interpolation of multi-view images is performed. For example, from a multi-view image of 5×5=25 viewpoints after frame interpolation and all depth maps, a viewpoint image between viewpoints is generated and interpolated. When generating a viewpoint image at a certain viewpoint (for example, coordinates (3, 3)), the surrounding decoded or frame interpolated viewpoint images (coordinates (1, 1), (1, 4), (4,1), (4,4)) as a reference image, interpolate the image based on the distance from the reference image (for example, using the reciprocal of the distance as the confidence level), and further use the depth map of each viewpoint. , generate a viewpoint image of the viewpoint.

In this embodiment, at this time, the reliability weight of the viewpoint image that is the reference image is added as a calculation element. In other words, the reliability based on the viewpoint image (coordinates (1, 4), (4, 1)) generated by frame interpolation is corrected to be smaller than the reliability based on the distance from the reference image, and the The reliability based on the viewpoint image (coordinates (1, 1), (4, 4)) decoded from the encoded data is corrected to a high value. For example, the reliability of interpolated viewpoint images is corrected with a weighting coefficient of 0.5, and the reliability of decoded viewpoint images is corrected with a weighting coefficient of 1.0. Note that the coefficient can be adjusted. Furthermore, when using a depth map, the reliability of a depth map generated by frame interpolation or viewpoint interpolation is set low, and the reliability of a depth map decoded from encoded data is set high. Based on the corrected reliability, interpolation processing is performed based on the reference image to generate and interpolate a multi-view image. Similar processing is performed for viewpoint images of other viewpoints.

In this way, multi-view images are interpolated, and the reproduced multi-view image group is output to the multi-view image element image conversion section 24.

The multi-view image element image conversion unit 24 converts the input multi-view images into a group of element images. The element image group can be created by converting multi-view images. That is, one element image is generated by extracting one pixel located at the same coordinate position from each viewpoint image forming a multi-view image and integrating the pixels while maintaining the overall arrangement of the multi-view images. For example, a 13×13 pixel element image is generated from one pixel of each viewpoint image of a multi-view image group captured by 13×13 cameras. By generating other element images in the same way, the multi-view image can be converted into an element image group in which 13×13 pixel element images are a set of the number of pixels of the viewpoint images. This is assumed to be the output image of the decoding device 20. This output image allows integral 3D video to be displayed.

Note that in this embodiment, the element image group is the output image on the premise that an integral 3D video is displayed based on the output image, but for example, in order to display a multi-view video, the multi-view image element The multi-view image after viewpoint interpolation may be used as the output image of the decoding device without providing the image conversion unit 24.

According to the decoding device of this embodiment, highly accurate viewpoint interpolation can be performed by using the reliability of images of each viewpoint. Furthermore, by making the distance between viewpoints relatively short after viewpoint thinning, it is possible to suppress deterioration in the image quality of the viewpoint interpolation image.

In the above embodiment, the configuration and operation of the encoding device 10 have been described, but the present invention is not limited to this, and may be configured as an encoding method for encoding a multi-view image. That is, the present invention may be configured as an encoding method that generates video encoded data from multi-view images according to the data flow shown in FIG. Further, although the configuration and operation of the decoding device 20 have been described, the present invention is not limited to this, and may be configured as a decoding method for decoding video encoded data. That is, the present invention may be configured as a decoding method that generates an output image of an elemental image group from encoded video data according to the data flow shown in FIG.

Note that a computer can be suitably used to function as the encoding device 10 or the decoding device 20 described above, and such a computer can be used to describe the processing content for realizing each function of the encoding device 10 or the decoding device 20. This can be achieved by storing a program in the storage section of the computer, and having the CPU of the computer read and execute this program. Note that this program can be recorded on a computer-readable recording medium.

Although the embodiments described above have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited to the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, it is possible to combine a plurality of configuration blocks described in the embodiments into one, or to divide one configuration block.

10 Encoding device 11 Viewpoint thinning section 12 Frame thinning section 13 Encoding section 20 Decoding device 21 Decoding section 22 Frame interpolation section 23 Viewpoint interpolation section 24 Multi-view image element image conversion section

Claims (7)

An encoding device that encodes a multi-view video and a depth map corresponding to the multi-view video, comprising:
a viewpoint thinning unit that thins out the viewpoints of the multi-view video and the depth map in each frame;
a frame thinning unit that thins out frames of the multi-view video and the depth map;
In an encoding device including an encoding unit that encodes a multi-view video that has undergone viewpoint thinning and frame thinning and a depth map,
The encoding device is characterized in that the frame thinning unit thins out either the viewpoint image or the depth map at each viewpoint of each frame, and switches the thinning target for each frame. The encoding device according to claim 1,
The frame thinning unit is characterized in that in each frame, one of the viewpoint image and the depth map of one viewpoint is thinned out, and the other of the viewpoint image and the depth map of at least one viewpoint adjacent to the one viewpoint is thinned out. Encoding device. The encoding device according to claim 1 or 2,
The encoding device is characterized in that the frame thinning unit thins out viewpoint images or depth maps in each frame subjected to viewpoint thinning so as to form a checkered pattern. The encoding device according to claim 1 or 2,
The encoding device is characterized in that the frame thinning unit thins out viewpoint images and depth maps in each frame subjected to viewpoint thinning so that they are alternately arranged in columns or rows. a decoding unit that decodes a multi-view video and a depth map from video encoded data;
a frame interpolation unit that interpolates thinned frames to the multi-view video and the depth map;
Each frame includes a viewpoint interpolation unit that interpolates the thinned-out viewpoint image and the depth map ,
setting the reliability of the viewpoint image and the depth map decoded from the video encoded data to a higher value than the reliability of the interpolated viewpoint image and the depth map;
The decoding device is characterized in that the viewpoint interpolation unit predicts the viewpoint image based on a reliability weight of the reference image . A program that causes a computer to function as the encoding device according to any one of claims 1 to 4. A program that causes a computer to function as the decoding device according to claim 5 .

Images