JP6949163B2 - Element image group generator, encoder, decoder, and program - Google Patents

Description

The present invention relates to an element image group generating device, a coding device, a decoding device, and a program.

The integral method has been proposed and studied as one of the imaging methods for 3D television that can be visually recognized from an arbitrary viewpoint. In the integral method, a group of lenses arranged on a plane or a spherical surface is used at the time of imaging. The arranged lens group is called a lens array. The individual lenses that make up the lens group are called element lenses. The output signal from the stereoscopic television camera (imaging apparatus) adopting this integral method is a signal in which the stereoscopic imaging signals obtained from the lens group are integrated.

It is known that the resolution of a stereoscopic image captured by the integral method is related to the number of pixels of the image sensor and the number of the above-mentioned element lenses. If the number of element lenses increases and the number of pixels of the image generated from one element lens (the image generated by each element lens is hereinafter referred to as "element image") is large, the stereoscopic image is taken. The image resolution also increases.
Similarly, in the display of a stereoscopic image by the integral method, a high-resolution natural stereoscopic image can be reproduced by increasing the number of element lenses and the number of pixels of the element image.

The range in which the stereoscopic image displayed by the integral method can be seen correctly is called the visual range. The visual field corresponds to the angle at which the element image is projected by the element lens, and is related to the focal length of the element lens and the element image size.

In order to acquire the element image group by the integral method, the subject is imaged by the image sensor through the lens array. In this method, the size of the subject to be photographed is limited, but since the element image group can be directly acquired by the image sensor, there is an advantage that the processing from shooting to display can be performed immediately. In order to shoot a wide range of spaces and subjects in an integral manner without any restrictions on the size of the subject, acquire multi-viewpoint images of the target object from multiple viewpoints and computer graphics the 3D model of the subject. (CG) A method of generating in space is used.

Patent Document 1 describes a method of acquiring an element image group by pseudo-photographing a three-dimensional model obtained from a multi-viewpoint image through a lens array arranged on CG.

Japanese Unexamined Patent Publication No. 2013-196532

Since the method described in Patent Document 1 requires a process of generating a three-dimensional model based on a multi-viewpoint image, an arithmetic unit having high processing power is required to generate an element image group. Another problem is that the arithmetic processing takes time. On the other hand, when the 3D model is generated, the change of the 3D model on the time axis can be captured, so that the area that cannot be displayed due to the shadow of the subject (this area is referred to as the "occlusion area"). ”) Can be interpolated.

Further, a method of generating an element image group from a multi-viewpoint image without generating a three-dimensional model is also known. In this method, depending on the structure of the lens array, it is necessary to generate an element image group including a component other than the element image, which is a component other than the element image. Is required, and there arises a problem that it takes time to generate the element image group.

Typical lens arrays for capturing and displaying integral stereoscopic images are a square array lens array and a delta array lens array.
The delta array is a so-called "bale-stacked" arrangement of circular element lenses.
That is, in the delta arrangement, circular element lenses of the same size are arranged so that the center points of adjacent element lenses form an equilateral triangle. When circular element lenses of the same size are arranged so as not to overlap each other, this delta arrangement is an arrangement in which the number of element lenses per unit area is the largest. By using this delta-arranged lens array, it is possible to increase the number of element lenses of the vertical component and improve the resolution of the displayed integral stereoscopic image.

When an element image group is generated using a lens array having a delta array, the generated element image group includes not only the components of the element image but also the components outside the element image. This is because there is a gap between the delta-arranged circular element lenses, and the light component passing through the gap is imaged.
Then, in order to generate this element image group from the multi-viewpoint image, it becomes necessary to extract a plurality of pixels constituting the multi-viewpoint image and perform interpolation processing from those pixel values. Therefore, there is a problem that the calculation becomes complicated and a lot of time is required to generate the element image group.

The present invention has been made based on the above-mentioned problem recognition, and an object of the present invention is to provide an element image group generating device capable of generating an element image group with a small amount of calculation or a calculation time.
Further, the present invention aims to provide a system (encoding device and decoding device) capable of highly efficiently encoding and decoding an element image group by applying the configuration of the element image group generating device described above. It is a thing.

[1] In order to solve the above problems, one aspect of the present invention is an element image group generation device that generates an element image group composed of element images for displaying a stereoscopic image by an integral method, and is a multi-viewpoint. The multi-viewpoint image group acquisition unit that acquires the multi-viewpoint image group formed by arranging the images viewed from the above, and the degree to which the components other than the element image are included in the pixels in the multi-viewpoint image group are included in the pixels. From the mask image storage unit that stores the mask image held as the pixel value and the pixel values of the pixels of the multi-viewpoint image group acquired by the multi-viewpoint image group acquisition unit, the mask image stored by the mask image storage unit It is characterized by including an element image external component subtraction unit that subtracts a pixel value of a pixel, and an image conversion unit that converts a multi-viewpoint image group output from the element image external component subtraction unit into an element image group. It is an element image group generator.

[2] Further, in one aspect of the present invention, an element image group acquisition unit that acquires an element image group composed of element images and the element image group acquired by the element image group acquisition unit are converted into a multi-viewpoint image group. An image conversion unit, a mask image storage unit that stores a mask image that stores a mask image in which a component other than the element image is included in a pixel in the multi-viewpoint image group as a pixel value of the pixel, and the mask image. By referring to the mask image stored in the storage unit, it is determined whether or not the component outside the element image is equal to or less than a predetermined value for the image of each viewpoint included in the multi-viewpoint image group output from the image conversion unit. , A multi-viewpoint image selection unit that selects and outputs only an image of a viewpoint whose component outside the element image is equal to or less than the predetermined value, and a viewpoint selected by the multi-viewpoint image selection unit in the multi-viewpoint image group. It is a coding apparatus including a multi-viewpoint image coding unit that encodes and outputs only an image.

[3] Further, in the coding device according to one aspect of the present invention, for each of the element images included in the element image group acquired by the element image group acquisition unit, the element image An image size conversion unit that converts an image size by inserting or deleting pixels in at least one of the vertical direction and the horizontal direction so that the pitch is an integral multiple of the pixel pitch is further provided. The conversion unit is characterized in that the element image group whose size has been converted by the image size conversion unit is converted into the multi-viewpoint image group.

[4] Further, in the coding apparatus according to one aspect of the present invention, when the image size conversion unit converts the image size, the pixel values of the pixels included in the element image before conversion are converted into the above-mentioned after conversion. The coding apparatus according to claim 3, wherein the pixel value is a pixel value of a pixel at a corresponding position included in an element image.

[5] Further, one aspect of the present invention is a decoding device that decodes a coded multi-viewpoint image, converts it into an element image group consisting of element images for displaying a stereoscopic image by an integral method, and outputs the image. There is a multi-viewpoint image decoding unit that decodes a code obtained by encoding a multi-viewpoint image that is an image of a plurality of viewpoints, and a viewpoint image that is insufficient based on the viewpoint image decoded by the multi-viewpoint image decoding unit. The insufficient multi-viewpoint image generation unit that generates the image of the above and outputs the multi-viewpoint image group consisting of the viewpoint image decoded by the multi-viewpoint image decoding unit and the generated image, and the components other than the element image are described above. A mask image storage unit that stores a mask image that holds the degree of inclusion in the pixels in the multi-viewpoint image group as a pixel value of the pixel, and a pixel of the multi-viewpoint image group output from the insufficient multi-viewpoint image generation unit. The element image includes an element image external component subtraction unit that subtracts the pixel value of the pixel of the mask image stored by the mask image storage unit from the pixel value, and a multi-viewpoint image group output from the element image external component subtraction unit. It is a decoding device characterized by including an image conversion unit for converting into a group.

[6] Further, the decoding device according to one aspect of the present invention reverses the conversion of the image size performed at the time of encoding for each of the element images included in the element image group output by the image conversion unit. It is characterized by further including an image size conversion unit that converts an image size by inserting or deleting pixels in at least one of the vertical direction and the horizontal direction so as to perform conversion.

[7] Further, in the decoding device according to one aspect of the present invention, when the image size conversion unit converts the image size, the pixel values of the pixels included in the element image before conversion are converted into the element after conversion. It is characterized in that it is the pixel value of the pixel at the corresponding position included in the image.

[8] Further, one aspect of the present invention is an element image group generation device for generating an element image group composed of element images for displaying a stereoscopic image by an integral method, and the computer is viewed from multiple viewpoints. The pixel value of the pixel value is the degree to which the multi-viewpoint image group acquisition unit that acquires the multi-viewpoint image group formed by arranging the images and the degree to which the components other than the element image are included in the pixels in the multi-viewpoint image group. From the pixel values of the mask image storage unit that stores the mask image to be held and the pixels of the multi-viewpoint image group acquired by the multi-viewpoint image group acquisition unit, the pixels of the pixels of the mask image stored by the mask image storage unit. It functions as an element image group generation device including an element image external component subtraction unit for subtracting a value and an image conversion unit for converting a multi-viewpoint image group output from the element image external component subtraction unit into an element image group. Program for.

[9] Further, in one aspect of the present invention, a computer is used to obtain a multi-viewpoint image of an element image group acquisition unit that acquires an element image group composed of element images and the element image group acquired by the element image group acquisition unit. An image conversion unit that converts into a group, a mask image storage unit that stores a mask image that holds the degree to which components other than the element image are included in the pixels in the multi-viewpoint image group as pixel values of the pixels, and a mask image storage unit. By referring to the mask image stored in the mask image storage unit, whether or not the component outside the element image is equal to or less than a predetermined value for the image of each viewpoint included in the multi-viewpoint image group output from the image conversion unit. Is selected by the multi-viewpoint image selection unit that selects and outputs only the image of the viewpoint whose component outside the element image is equal to or less than the predetermined value, and the multi-viewpoint image selection unit of the multi-viewpoint image group. This is a program for functioning as a coding device including a multi-viewpoint image coding unit that encodes and outputs only an image of a viewpoint.

[10] Further, in one aspect of the present invention, a computer decodes a coded multi-viewpoint image, converts it into an element image group composed of element images for displaying a stereoscopic image by an integral method, and outputs the image. Based on a multi-viewpoint image decoding unit that is a decoding device and decodes a code obtained by encoding a multi-viewpoint image that is an image of a plurality of viewpoints, and a viewpoint image decoded by the multi-viewpoint image decoding unit. A missing multi-viewpoint image generator that generates an image of the missing viewpoint and outputs a multi-viewpoint image group consisting of the viewpoint image decoded by the multi-viewpoint image decoding unit and the generated image, and a non-deficient multi-viewpoint image generation unit outside the element image. A mask image storage unit that stores a mask image that holds the degree to which a component is included in a pixel in the multi-viewpoint image group as a pixel value of the pixel, and the multi-viewpoint image group output from the insufficient multi-viewpoint image generation unit. The element image external component subtraction unit that subtracts the pixel value of the mask image pixel stored in the mask image storage unit from the pixel value of the pixel, and the multi-viewpoint image group output from the element image external component subtraction unit. This is a program for functioning as a decoding device including an image conversion unit for converting into the element image group.

According to the present invention, it is possible to generate an element image by a process of subtracting a pixel value of a mask image. That is, it is possible to reduce the processing amount or processing time for generating the element image.

It is a block diagram which shows the schematic functional structure of the element image group generating apparatus by 1st Embodiment of this invention. It is the schematic for demonstrating the arrangement of the lens of the lens array by the delta arrangement, and the size conversion ratio in the image size conversion part by the same embodiment. It is a block diagram which shows the schematic functional structure of the mask image generation apparatus for generating the mask image used by the element image group generating apparatus by this embodiment. It is the schematic which shows an example of the image in the process of generating the mask image used in the same embodiment. It is the schematic which shows an example of the mask image used in the same embodiment. The mask image is generated by the above-mentioned mask image generator. It is a schematic diagram which shows the example of the multi-viewpoint image group acquired from the outside by the multi-viewpoint image group acquisition part of the element image group generation apparatus by the same embodiment, and was resized by the image size conversion part. It is a schematic diagram which shows the example of the multi-viewpoint image group output by the element image external component subtraction part of the element image group generation apparatus by the same embodiment. It is a block diagram which shows the schematic functional structure of the element image group generating apparatus by 2nd Embodiment. It is a block diagram which shows the schematic functional structure of the coding apparatus and decoding apparatus according to 3rd Embodiment. It is a schematic diagram which shows the example of the result of selection of viewpoints in the multi-viewpoint image selection part by the same embodiment. It is the schematic which shows the example of the image in the process of restoring a multi-viewpoint image on the decoding apparatus side by the same embodiment. It is a schematic diagram which shows the multi-viewpoint image group of the result of restoration on the decoding apparatus side by the same embodiment. It is a schematic diagram which shows the example of the multi-viewpoint image group obtained when the image size conversion is not performed. It is a schematic diagram which shows the example of the multi-viewpoint image group obtained when the image size conversion processing is performed. It is a schematic diagram for demonstrating the positional relationship of the pixel before and after the conversion of an image size by the modification of embodiment. It is a schematic diagram which shows the reference relation of the pixel value before and after the return in the process which the image size conversion unit performs the size conversion by the modification. It is a schematic diagram which shows the multi-viewpoint image group generated based on the element image group which was resized by the modification.

Next, an embodiment of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram showing a schematic functional configuration of an element image group generating device according to the present embodiment. The element image group generation device 1 generates an element image group composed of element images for displaying a stereoscopic image by an integral method. As shown in the figure, the element image group generation device 1 includes a mask image storage unit 10, a multi-viewpoint image group acquisition unit 21, an image size conversion unit 22, an element image external component subtraction unit 23, and an image conversion unit 31. , An image size conversion unit 32 and an element image group output unit 33 are included.
In the following, the pixel value in the image is at least a predetermined minimum value and at least a predetermined maximum value. Here, the minimum value is zero. When the image is composed of a plurality of channels (for example, it is composed of the three primary colors of R (red), G (green), and B (blue)), the pixel value of each channel is equal to or more than the minimum value and described above. It is below the maximum value.

The mask image storage unit 10 stores the mask image. The mask image stored in the mask image storage unit 10 is generated in advance based on the structure of the lens array used for displaying the stereoscopic image. The mask image is data having a pixel value for each pixel. The pixel value of each pixel of the mask image represents the degree of the amount of the component outside the element image in that pixel. When the pixel value of the mask image is the minimum value (zero), it means that the pixel has no component outside the element image. When the pixel value of the mask image is the maximum value, it means that the pixel has the maximum component outside the element image. The mask image stored in the mask image storage unit 10 is generated on the premise that a lens array based on a delta array is used. That is, the mask image storage unit 10 stores a mask image that retains the degree to which components outside the element image (components outside the element image) are included in the pixels in the multi-viewpoint image group as pixel values of the pixels. .. The method of generating the mask image will be described later.

The multi-viewpoint image group acquisition unit 21 acquires a multi-viewpoint image from the outside. The multi-viewpoint image group is an image obtained by capturing a subject with a multi-viewpoint camera. That is, the multi-viewpoint image group acquisition unit 21 acquires a multi-viewpoint image group formed by arranging images viewed from multiple viewpoints. For example, when there are a total of 64 viewpoints of 8 vertical × 8 horizontal, multi-viewpoint images from those viewpoints can be captured by 64 cameras arranged in 8 vertical × 8 horizontal. It is also possible to take an image with a camera having a smaller number of viewpoints and obtain a multi-viewpoint image by interpolation. For example, using images captured by 32 cameras, images of 32 viewpoints different from those 32 viewpoints can be generated by interpolation to obtain a total of 64 viewpoints of multi-viewpoint images. When interpolating an image, interpolation interpolation is used. Further, the multi-viewpoint image group acquisition unit 21 may obtain the multi-viewpoint image directly from those cameras, or may acquire the image data previously captured by those cameras from the outside.

The image size conversion unit 22 converts the size of the multi-viewpoint image acquired by the multi-viewpoint image group acquisition unit 21. Specifically, the image size conversion unit 22 changes the size of the multi-viewpoint image so as to match the aspect ratio of the lens array based on the delta arrangement. More specifically, the image size conversion unit 22 converts the image size so that the number of pixels in the vertical direction of the multi-viewpoint image is (2 / SQRT (3)) times (≈1.1547 times). That is, the image size conversion unit 22 converts pixels in at least one of the vertical direction and the horizontal direction so that the pitch of the element images is an integral multiple of the pixel pitch for each of the element images included in the element image group. Convert the image size by inserting or deleting. Here, "SQRT ()" represents a square root. The image size conversion unit 22 does not change the image size in the horizontal direction. At this time, the image size conversion unit 22 obtains the pixel value of each pixel after the conversion of the image size by the interpolation calculation of the pixel value before the conversion. Instead of using the interpolation calculation, the pixel value of each pixel after the size conversion may be obtained by the method of the modification 3 described later.
The pixel ratio in the image size conversion process will be supplementarily described later with reference to another drawing.

The element image external component subtraction unit 23 performs a process of subtracting the mask image read from the mask image storage unit 10 from the multi-viewpoint image (after image size conversion) output by the image size conversion unit 22. The element image external component subtraction unit 23 subtracts the pixel value for each pixel of both of these images. That is, the element image external component subtraction unit 23 subtracts the pixel values of the mask image pixels stored in the mask image storage unit 10 from the pixel values of the pixels of the multi-viewpoint image group acquired by the multi-viewpoint image group acquisition unit 21. .. The element image external component subtraction unit 23 subtracts the pixel values of the corresponding pixels of both images. The minimum pixel value is zero. If the value of the result of subtraction is negative, the pixel value of that pixel is set to zero.

The image conversion unit 31 converts the multi-viewpoint image group output from the element image external component subtraction unit 23 into an element image group.
The process itself of converting the multi-viewpoint image group into the element image group can be performed by the existing technique. Based on the multi-viewpoint image group in which the pixels for each viewpoint are arranged together, the image conversion unit 31 generates an element image group in which the pixels for each element lens are arranged together by rearranging the pixels. do.

The image size conversion unit 32 converts the size of the image output from the image conversion unit 31. The size conversion process in the image size conversion unit 32 is the inverse conversion of the size conversion process in the image size conversion unit 22 described above. Specifically, the image size conversion unit 32 converts the image size so that the number of pixels in the vertical direction of the image is (SQRT (3) / 2) times (≈0.8660 times). That is, the image size conversion unit 32 shifts pixels in at least one of the vertical direction and the horizontal direction so that the pitch of the element images is an integral multiple of the pixel pitch for each of the element images included in the element image group. Convert the image size by inserting or deleting. The image size conversion unit 32 does not change the image size in the horizontal direction. Here, too, the image size conversion unit 32 obtains the pixel value of each pixel after the conversion of the image size by the interpolation calculation of the pixel value before the conversion. Instead of using the interpolation calculation, the pixel value of each pixel after the size conversion may be obtained by the method of the modification 3 described later.
By the above size conversion process by the image size conversion unit 32, the pitch of the element lens becomes an integral multiple of the pixel pitch.
Then, the element image group output unit 33 outputs the element image group obtained by the above processing.

FIG. 2 is a diagram for explaining the size conversion ratio in each of the image size conversion unit 22 and the image size conversion unit 32. The figure is a plan view of a part of a large number of element lenses arranged in a delta arrangement. In the figure, only four element lenses out of a large number of element lenses are shown, and other element lenses are omitted. The triangles shown in the figure are triangles having the center point of each circular element lens as an apex, and are equilateral triangles.
As shown in the figure, in the case of the delta arrangement, the pitch of the element lenses in the horizontal direction and the pitch of the element lenses in the vertical direction are 2 pairs SQRT (3). Then, based on this ratio, the conversion ratio in the image size conversion unit 22 and the image size conversion unit 32 is determined.

Next, a method of generating a mask image to be used in the element image group generating device 1 will be described.
FIG. 3 is a block diagram showing a schematic functional configuration of the mask image generator. As shown in the figure, the mask image generation device 40 includes a three-dimensional model storage unit 41, an element image group generation unit 42, an image size conversion unit 43, an image conversion unit 44, an inversion unit 45, and a mask image output unit. It is configured to include 46 and. In the method described here, the mask image generation device 40 generates an element image group based on the three-dimensional model, and converts the element image group into a multi-viewpoint image group. The functions of each part are as described below.

The 3D model storage unit 41 stores a 3D model for computer graphics. Here, the three-dimensional model stored in the three-dimensional model storage unit 41 for generating the mask image is a 100% white plane on the lens array for use in displaying the three-dimensional image. When the 3D model is "100% white", the pixel value obtained when the 3D model is drawn directly is the maximum value (when expressed in the three primary colors of RGB, the pixels in each channel of RGB. The value is the maximum value).

The element image group generating unit 42 is an element image when a three-dimensional model stored in the three-dimensional model storage unit 41 is used as a subject according to the structure (position, number, size, focal length, etc.) of the given lens array. Generate a group.

The image size conversion unit 43 performs a process of changing the image size of the element image group generated by the element image group generation unit 42. Here, specifically, the image size conversion unit 43 sets the change ratio of the image size in the vertical direction to (2 / SQRT (3)) times (≈1.1547 times). The image size conversion unit 43 does not change the image size in the horizontal direction. The image size conversion unit 43 obtains the pixel value of each pixel after conversion of the image size by interpolation calculation of the pixel value before conversion. As described with reference to FIG. 2, the change ratio of the image size is determined by the lens pitch of the element lenses constituting the lens array of the delta arrangement. By changing the image size in this way, the image size conversion unit 43 makes the relationship between the lens pitch of the element lens and the pixel pitch of the display device an integer ratio.
The image size conversion unit 43 performs the above size conversion process in order to match the size of the multi-viewpoint image to be processed (the multi-viewpoint image to be subtracted from the elements other than the element image represented by the mask image). .. If the sizes are already matched, the size conversion process by the image size conversion unit 43 is omitted.

The image conversion unit 44 converts the element image group (image size changed) output from the image size conversion unit 43 into a multi-viewpoint image group. The image output from the image conversion unit 44 will be described later with reference to the drawings.
The process itself of converting the element image group into the multi-viewpoint image group can be performed by the existing technique. Based on the element image group in which the pixels for each element lens are arranged together, the image conversion unit 44 generates a multi-viewpoint image group in which the pixels for each viewpoint are arranged together by rearranging the pixels. do.

The inversion unit 45 performs a negative / positive inversion process on the multi-viewpoint image group output from the image conversion unit 44. The inversion unit 45 performs negative / positive inversion for each pixel of the input image. In the negative / positive inversion process, the mathematical formula is expressed as output pixel value = maximum pixel value-input pixel value. By the processing of the inversion unit 45, the white pixels are converted to black, and the black pixels are converted to white. In addition, the pixels of the intermediate gradation are also inverted in the same manner. The image output from the inversion unit 45 will be described later with reference to the drawings. The image obtained as a result of the processing by the inversion unit 45 is a mask image.
The mask image output unit 46 outputs the mask image generated by the inversion unit 45 to the outside.

For a lens array defined by a predetermined parameter, the mask image can be used repeatedly for the same lens array by performing the calculation for obtaining the element image only once and storing the generated mask image. .. The mask image storage unit 10 stores the mask image.

FIG. 4 is a schematic view showing an example of an image output from the image conversion unit 44 in the process of generating a mask image. The illustrated image is an image after being converted from an element image group to a multi-viewpoint image group by the image conversion unit 44. In the image shown here, the original three-dimensional model (100% white plane) is reflected in the white pixels (pixels having a large pixel value). Further, in this image, black pixels (pixels having a small pixel value) reflect components outside the element image. The half-gradation pixel represents the degree of the component outside the element image (the smaller the pixel value, the higher the degree of the component outside the element image) according to the pixel value. That is, the image shown in the figure shows how the components outside the element image are distributed after being converted from the element image group to the multi-viewpoint image group.

FIG. 5 is a schematic view showing an example of a mask image generated by the mask image generator. The illustrated image is an image obtained as a result of the negative / positive reversal processing of the image shown in FIG. 4 by the reversing unit 45. That is, in the image shown in FIG. 5, white pixels (pixels having a large pixel value) are pixels corresponding to components outside the element image. It can also be said that the larger the pixel value, the greater the degree to which the pixel contains components outside the element image.

Instead of generating the mask image by the mask image generation device 40 shown in FIG. 3, the mask image may be generated by the following method. That is, a pure white object (for example, a pure white flat plate) is photographed with a light field camera, an IP (integral photography) camera, or the like. Then, a mask image is generated by inverting the brightness of the obtained image.

Next, an example of a multi-viewpoint image group processed by the element image group generator will be described.
FIG. 6 is a schematic view showing an example of a multi-viewpoint image group acquired by the multi-viewpoint image group acquisition unit 21 of the element image group generation device 1 from the outside and whose size is changed by the image size conversion unit 22. The illustrated image is a multi-viewpoint image obtained from a total of 64 viewpoints (8 vertical × 8 horizontal).

FIG. 7 is a schematic view showing an example of a multi-viewpoint image group output by the element image external component subtraction unit 23 of the element image group generation device 1. That is, the mask image shown in FIG. 5 is stored in the mask image storage unit 10 in advance, and the mask image (FIG. 5) read from the mask image storage unit 10 is subtracted from the multi-viewpoint image shown in FIG. The image obtained as a result of this is the multi-viewpoint image shown in FIG. Comparing the image of FIG. 6 (the image before subtracting the mask image) and the image of FIG. 7 (the image after subtracting the mask image), the image mainly corresponding to the viewpoint of the peripheral portion is based on the mask image. The effect of subtraction is relatively strong. This corresponds to the distribution of whitish pixels (pixels having a relatively large pixel value) in the mask image shown in FIG.

As described above, according to the element image group generating device 1 according to the present embodiment, since the mask image calculated and stored in advance is subtracted from the multi-view image, the multi-view image in which the components outside the element image are suppressed is obtained. It can be generated (output from the element image external component subtraction unit 23). Then, the multi-viewpoint image group output from the element image external component subtraction unit 23 is converted into an element image group by the image conversion unit 31, and then the image size conversion unit 32 reconverts the image size (image size conversion unit 22). By performing the inverse conversion of the size conversion in (1), a desired element image group can be generated.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The matters already described in the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.
FIG. 8 is a block diagram showing a schematic functional configuration of the element image group generating device according to the present embodiment. As shown in the figure, the element image group generation device 2 includes a mask image storage unit 11, a multi-viewpoint image group acquisition unit 21, an element image external component subtraction unit 23, an image conversion unit 31, and an element image group output unit 33. It is composed including and.

In the present embodiment, the mask image storage unit 11 stores the mask image, which is the same as the mask image storage unit 10 in the previous embodiment. However, the mask image storage unit 11 according to the present embodiment is characterized in that it stores a mask image whose size has been changed according to the structure of the lens array used. That is, for the mask image stored by the mask image storage unit 11, the horizontal size of the image is increased by 1 and the vertical size is increased by (SQRT (3) / 2) times (≈0.8660 times) in advance. It has been resized.
Therefore, the element image external component subtraction unit 23 according to the present embodiment performs a process of subtracting the mask image stored in the mask image storage unit 11 from the multi-viewpoint image acquired by the multi-viewpoint image group acquisition unit 21. .. However, for each pixel, if the result of subtracting the pixel value becomes negative, the pixel value of that pixel is set to zero.

Therefore, in the present embodiment, the size conversion processing by the image size conversion unit 22 and the image size conversion unit 32 that the element image group generating device 1 according to the first embodiment has is not required. Therefore, in the present embodiment, it is possible to generate the element image group at a higher speed.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The matters already described up to the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.
FIG. 9 is a block diagram showing a schematic functional configuration of a coding device and a decoding device according to the present embodiment. The coding device 3 is a device that converts an input element image group into a multi-viewpoint image and encodes it. The code generated by the coding device 3 is passed to the decoding device 4 by transmission or via a recording medium. The decoding device 4 is a device that acquires the code output from the coding device 3, decodes it, converts it into an element image group, and outputs the code. That is, the decoding device 4 decodes the encoded multi-viewpoint image, converts it into an element image group composed of element images for displaying a stereoscopic image by the integral method, and outputs the image.

As shown in the figure, the coding device 3 includes a mask image storage unit 13, an element image group acquisition unit 51, an image size conversion unit 52, an image conversion unit 53, a multi-viewpoint image selection unit 54, and a multi-viewpoint image. It is configured to include a coding unit 55.
Further, the decoding device 4 includes a mask image storage unit 14, a multi-viewpoint image decoding unit 61, a lacking multi-viewpoint image generation unit 62, an element image external component subtraction unit 63, an image conversion unit 64, and an image size conversion unit. 65 and an element image group output unit 66 are included.

The functions of each part of the coding device 3 and the decoding device 4 and the flow of processing by these devices will be described below.
First, the element image group acquisition unit 51 of the coding device 3 acquires the element image group from the outside. The element image group is an image in which a plurality of element images are arranged. The element image group acquisition unit 51 acquires, for example, an element image group imaged by using an element lens and an image sensor. Further, the element image group acquisition unit 51 may acquire an element image group as a still image, or may acquire an image of an element image group configured as a time-series image.
The element image group acquired by the element image group acquisition unit 51 includes components outside the element image.

The image size conversion unit 52 converts the size of the image in the same manner as the image size conversion unit 22 in the first embodiment. Specifically, the image size conversion unit 52 converts the size of the image output from the element image group acquisition unit 51 and outputs the image.
The image conversion unit 53 converts the element image group output from the image size conversion unit 52 into a multi-viewpoint image group. That is, the element image group acquired by the element image group acquisition unit 51 is converted into the multi-viewpoint image group.

Similar to the mask image storage unit 10 in the first embodiment, the mask image storage unit 13 stores in advance a mask image representing the degree of the component outside the element image. That is, the mask image storage unit 13 stores a mask image that retains the degree to which components outside the element image are included in the pixels in the multi-viewpoint image group as pixel values of the pixels. The method of generating the mask image is as described as the function of the mask image generation device 40 in the first embodiment.

The multi-viewpoint image selection unit 54 excludes the image from the viewpoint including the component outside the element image from the multi-viewpoint image group output by the image conversion unit 53, and selects only the image from the viewpoint not including the component outside the element image. do. At this time, the multi-viewpoint image selection unit 54 selects whether or not the image from a certain viewpoint contains components other than the element image, based on the value of each pixel of the mask image read from the mask image storage unit 13.
In the multi-viewpoint image selection unit 54, in order to determine whether or not the image of a certain viewpoint contains components other than the element image, for example, a threshold value appropriately determined for the pixel value of the mask image is used. .. Then, the multi-viewpoint image selection unit 54 calculates the average value of the pixel values of each pixel included in the region of the image of a specific viewpoint in the multi-viewpoint image group in the corresponding mask image. Then, if the calculated average value is equal to or higher than the above threshold value, the multi-viewpoint image selection unit 54 determines that the viewpoint image includes components outside the element image. On the contrary, if the calculated average value is less than the above threshold value, the multi-viewpoint image selection unit 54 determines that the image of the viewpoint does not include the component outside the element image.
That is, the multi-viewpoint image selection unit 54 refers to the mask image stored in the mask image storage unit 13 to display element images for the images of each viewpoint included in the multi-viewpoint image group output from the image conversion unit 53. It is determined whether or not the external component is equal to or less than a predetermined value, and only the image of the viewpoint in which the external component of the element image is equal to or less than the predetermined value is selected and output.

The multi-viewpoint image coding unit 55 encodes and outputs only the image selected by the multi-viewpoint image selection unit 54 from the multi-viewpoint image group output from the image conversion unit 53. That is, the multi-viewpoint image coding unit 55 encodes and outputs only the image that does not include the component outside the element image. That is, the multi-viewpoint image coding unit 55 encodes and outputs only the image of the viewpoint selected by the multi-viewpoint image selection unit 54 in the multi-viewpoint image group. The image (video) coding itself can be performed by appropriately applying the existing technology.

As described above, the coding apparatus 3 according to the present embodiment excludes the multi-viewpoint image including the component other than the element image from the object of coding by the determination by the multi-viewpoint image selection unit 54. That is, the coding device 3 encodes the multi-viewpoint image including only the element image components, and outputs a code that does not include the multi-viewpoint image including the components outside the element image.

On the decoding device 4, the multi-viewpoint image decoding unit 61 decodes the multi-viewpoint image output from the coding device 3. As described above, since the multi-viewpoint image selection unit 54 of the coding apparatus 3 selects the multi-viewpoint image, the image decoded by the multi-viewpoint image decoding unit 61 lacks the image of a part of the viewpoint. .. That is, the multi-viewpoint image decoding unit 61 decodes the code obtained by encoding the multi-viewpoint image which is an image of a plurality of viewpoints.

The deficient multi-viewpoint image generation unit 62 generates the deficient multi-viewpoint image (that is, an image of a viewpoint excluded by the multi-viewpoint image selection unit 54 as an image including components outside the element image). Then, the deficient multi-viewpoint image generation unit 62 outputs the image group decoded by the multi-viewpoint image decoding unit 61 and the deficient viewpoint image group generated by itself as one multi-viewpoint image group. That is, the missing multi-viewpoint image generation unit 62 generates an image of the missing viewpoint based on the image of the viewpoint decoded by the multi-viewpoint image decoding unit 61, and together with the image of the viewpoint decoded by the multi-viewpoint image decoding unit 61. Outputs a multi-viewpoint image group consisting of the generated image.
Specifically, the missing multi-viewpoint image generation unit 62 generates an image of the missing viewpoint based on the multi-viewpoint image group decoded by the multi-viewpoint image decoding unit 61. As a more specific method, for example, the missing multi-viewpoint image generation unit 62 duplicates an image located at a viewpoint closest to the missing viewpoint image (an image decoded by the multi-viewpoint image decoding unit 61). By doing so, an image of the missing viewpoint is generated. Alternatively, for example, the missing multi-viewpoint image generation unit 62 performs interpolation processing of the viewpoint using the image of the viewpoint around the image of the missing viewpoint (the image decoded by the multi-viewpoint image decoding unit 61). By doing so, an image of the missing viewpoint is generated.

Similar to the mask image storage unit 13, the mask image storage unit 14 stores in advance a mask image indicating the degree of the component outside the element image. That is, the mask image storage unit 14 stores a mask image that retains the degree to which components outside the element image are included in the pixels in the multi-viewpoint image group as pixel values of the pixels.
The element image external component subtraction unit 63 performs a process of subtracting the mask image read from the mask image storage unit 14 from the multi-viewpoint image group output by the missing multi-viewpoint image generation unit 62. Specifically, the element image external component subtraction unit 63 subtracts the pixel value of each pixel included in the mask image from the pixel value of each pixel included in the multi-viewpoint image group. That is, the element image external component subtraction unit 63 subtracts the pixel values of the mask image pixels stored in the mask image storage unit 14 from the pixel values of the pixels of the multi-viewpoint image group output from the insufficient multi-viewpoint image generation unit 62. do. If the result of the subtraction is negative for each pixel, the pixel value of that pixel is set to zero. This process is a process in which the element image external component subtraction unit 63 subtracts the element image external component from the multi-viewpoint image group output by the missing multi-viewpoint image generation unit 62.

The image conversion unit 64 converts the multi-viewpoint image output from the element image external component subtraction unit 63 into an element image. The technique for converting a multi-viewpoint image group into an element image group has already been described.
The image size conversion unit 65 converts the image size of the element image output from the image conversion unit 64. The image size conversion process by the image size conversion unit 65 is an inverse conversion process of the conversion process in the image size conversion unit 52 of the coding apparatus 3. That is, the image size conversion unit 65 performs the inverse conversion of the image size conversion performed at the time of encoding for each of the element images included in the element image group output by the image conversion unit 64 in the vertical direction. Alternatively, the image size is transformed by inserting or removing pixels in at least one of the horizontal directions. That is, the image size conversion unit 65 converts the size of the image in the same manner as the image size conversion unit 32 in the first embodiment.
The element image group output unit 66 outputs the element image that has been size-converted by the image size conversion unit 65.

Next, a specific processing example in this embodiment will be described with reference to simulated image data.
FIG. 10 is a schematic view showing an example of the result of selection of viewpoints in the multi-viewpoint image selection unit 54. The image illustrated here is a multi-viewpoint image, and includes a total of 64 viewpoint images of 8 vertical (8 rows) × 8 horizontal (8 columns). As described above, the multi-viewpoint image selection unit 54 excludes the image of the viewpoint including the component outside the element image by referring to the mask image storage unit 13, and outputs only the image of the viewpoint including the component outside the element image. In the example shown in the figure, the multi-viewpoint image selection unit 54 excludes the image of the viewpoint outside the frame line as the image of the component outside the element image, and outputs the image of the viewpoint inside the frame line as the image of the viewpoint to be output. Select. Specifically, the multi-viewpoint image selection unit 54 includes images of the viewpoints of all columns in the first row and images of the viewpoints of the first column, the second column, the seventh column, and the eighth column in the second row. , The image of the viewpoint of the first column and the eighth column in each row from the third row to the seventh row, and the image of the viewpoint of the first column, the second column, the seventh column, and the eighth column of the eighth row. , Exclude as an image of components outside the element image. Further, the multi-viewpoint image selection unit 54 determines the images of the viewpoints from the third column to the sixth column in the second row and the viewpoints from the second column to the seventh column in each row from the third row to the seventh row. The image and the image of the viewpoint from the third column to the sixth column of the eighth row are selected as the image to be output.
Then, the multi-viewpoint image coding unit 55 encodes and outputs only the image group of the viewpoint selected by the multi-viewpoint image selection unit 54 (the image group that does not include the component outside the element image).

As illustrated with reference to the figure, in the coding apparatus 3 according to the present embodiment, the multi-viewpoint image selection unit 54 excludes the image of the component other than the element image, so that the coding efficiency can be improved.

FIG. 11 is a schematic view showing an example of an image in a process of restoring a multi-viewpoint image on the decoding device side. As shown in the figure, this multi-viewpoint image group does not include the viewpoint image group excluded by the multi-viewpoint image selection unit 54 of the encoding device 3. Then, as described above, the missing multi-viewpoint image generation unit 62 of the decoding device 4 complements the multi-viewpoint image group shown in the figure by generating an image of the missing viewpoint. In the illustrated example, specifically, the missing multi-viewpoint image generation unit 62 generates a missing multi-viewpoint image according to the arrows in the figure. That is, the images of the viewpoints of the first row, the second column, and the third column are generated based on the images of the viewpoints of the second row and the third column. An image of the viewpoint of the first row and the fourth column is generated based on the image of the viewpoint of the second row and the fourth column. An image of the viewpoint of the first row and the fifth column is generated based on the image of the viewpoint of the second row and the fifth column. Based on the image of the viewpoint of the second row and the sixth column, the image of the viewpoint of the first row, the sixth column and the seventh column is generated. Based on the image of the viewpoint of the third row and the second column, the image of the viewpoint of the second row, the first column and the third row and the first column is generated. Based on the image of the viewpoint of the third row and the third column, the image of the viewpoint of the first row, the first column and the second row and the second column is generated. Based on the image of the viewpoint of the third row and the sixth column, the image of the viewpoint of the first row, the eighth column and the second row and the seventh column is generated. Based on the image of the viewpoint of the third row and the seventh column, the image of the viewpoint of the second row and the eighth column and the third row and the eighth column is generated. Based on the image of the viewpoint of the second column of each of the fourth row to the sixth row, the image of the viewpoint of the first column of the row is generated. Based on the image of the viewpoint of the 7th column of each of the 4th to 6th rows, the image of the viewpoint of the 8th column of the row is generated. Based on the image of the viewpoint of the 7th row and the 2nd column, the image of the viewpoint of the 7th row and the 1st column, the 8th row and the 1st column, and the 8th row and the 2nd column is generated. Based on the image of the viewpoint of the 7th row and the 7th column, the image of the viewpoint of the 7th row and the 8th column, the 8th row and the 7th column, and the 8th row and the 8th column is generated.
The reference pattern of the viewpoint image shown here as a method for restoring the multi-view image is only an example, and the image of the viewpoint that is lacking due to duplication or interpolation interpolation processing by another reference pattern. May be generated.

FIG. 12 is a schematic view showing a group of multi-viewpoint images as a result of restoration on the decoding device side. As shown in the figure, the image of the viewpoint (see FIG. 10) excluded by the multi-viewpoint image selection unit 54 of the coding device 3 is restored by the lacking multi-viewpoint image generation unit 62 of the decoding device 4. Of course, the original multi-viewpoint image before encoding (the image at the stage output from the image conversion unit 53) and the image after restoration (the image at the stage output from the insufficient multi-viewpoint image generation unit 62) are the same. is not it. However, the viewpoint image restored by the insufficient multi-viewpoint image generation unit 62 is considered to be practically sufficient as an approximate image. Further, the restored image of the viewpoint is an image of the viewpoint that originally contained the component outside the element image, and is the target of subtraction (the image to be subtracted) by the processing in the element image external component subtraction unit 63.

As described above, in the present embodiment, the size of the coded (compressed) data can be reduced by reducing the number of multi-viewpoint images to be coded by the multi-viewpoint image coding unit 55. can. Further, since the viewpoint image including the components other than the element image has many high-frequency components, it is not necessary to encode the image of such a viewpoint, so that highly efficient coding can be performed. Therefore, when the encoded data is transmitted by a communication line or the like, the transmission efficiency can be improved. Further, when the encoded data is recorded on a recording medium or the like, the recording efficiency can be improved.

Next, a modified example of the embodiment will be described.
[Modification 1]
In the third embodiment, the mask image generated in advance is stored in the mask image storage unit 13 on the encoding device 3 side and the mask image storage unit 14 on the decoding device 4.
As a modification, the mask image may also be encoded on the coding device 3 side, and the encoded mask image may be passed to the decoding device 4. On the decoding device 4, the mask image is decoded from the acquired code, and the mask image is used in the element image external component subtraction unit 63.

[Modification 2]
In the above embodiment, the image size conversion unit converts the image size so that the pixel pitch matches the sampling points of the lenses constituting the lens array.
As a modification, the size conversion process for matching the pixel pitch to the sampling points of the lenses constituting the lens array may be omitted.
Even when the size conversion process is omitted, the process of subtracting the components outside the element image based on the mask image (the element image outside component subtraction unit 23 of the first embodiment) and the process of selecting the multi-viewpoint image (the first). The effect of the multi-viewpoint image selection unit 54) of the second embodiment can be obtained.
However, as described below, noise can be reduced and coding efficiency can be improved by performing a process of converting the image size.

FIG. 13 is a schematic view showing an example of a multi-viewpoint image group obtained when the image size conversion process is not performed. The illustrated example is a multi-viewpoint image group consisting of a total of 64 viewpoints, which are 8 vertical × 8 horizontal.
FIG. 14 is a schematic view showing an example of a multi-viewpoint image group obtained when the image size conversion process is performed. The illustrated example is a multi-viewpoint image group consisting of a total of 64 viewpoints, which are 8 vertical × 8 horizontal.
Comparing the image shown in FIG. 13 with the image shown in FIG. 14, it can be seen that a large amount of random noise is generated in the multi-viewpoint image group when the image size is not changed. This is because a delta-arranged lens array is used, and when the size conversion process is not performed (in the case of FIG. 13), the pixel pitch of the monitor to be displayed and the sampling points of the lenses constituting the lens array are set. Due to inconsistency. That is, if these two do not match, the components outside the element image are diffused and displayed as random noise. On the other hand, when the size conversion process is performed (in the case of FIG. 14), the pixel pitch of the displayed monitor and the sampling points of the lenses constituting the lens array match (the size is converted in that way), so that the result is obtained. In the multi-viewpoint image group, the components outside the element image are concentrated at a specific position.

When the multi-viewpoint image coding unit 55 encodes an image, the larger the size of the block, which is the basic unit of the coding process, the more efficiently the coding can be performed. Therefore, in the multi-viewpoint image group (FIG. 14) when the size conversion process is performed, the components outside the element image tend to be locally concentrated, and therefore, from the multi-viewpoint image group (FIG. 13) when the size conversion process is not performed. Can also perform highly efficient coding.

[Modification 3]
In each of the above-described embodiments, the image size conversion unit (reference numerals 22, 32, 43, 52, 65) calculates the pixel value after size conversion by interpolating the pixel values of adjacent pixels or neighboring pixels. And said.
In this modification, the image size conversion unit does not perform the above interpolation processing when converting the image size, but instead refers to the pixel values of adjacent pixels and copies the pixel values as they are. That is, when the image size is converted, the image size conversion unit sets the pixel value of the pixel included in the element image before conversion as the pixel value of the pixel at the corresponding position included in the element image after conversion. The process of referring to the pixel value and copying the pixel value as it is can be applied to both the image size conversion in the direction of decreasing the number of pixels and the image size conversion in the direction of increasing the number of pixels. That is, when the image size is converted in the direction in which the number of pixels decreases, there may be pixel values that are discarded without being referenced. Further, when the image size is converted in the direction of increasing the number of pixels, a portion where a plurality of pixels having the same pixel value are generated may occur at least in a part of the image.
When performing such image size conversion, if there is information on the pixel position added when the number of pixels increases, the conversion in the opposite direction (conversion that reduces the number of pixels) can be regarded as reversible conversion. can.

Here, the calculation of the pixel position in this modification will be described.
FIG. 15 is a schematic diagram for explaining the positional relationship of the pixels before and after the conversion of the image size. FIG. 3A is a plan view of the arrangement of the element images before the image size conversion.
FIG. 3B is a plan view of the arrangement of the element images after the image size conversion. In each of the figures (a) and (b), the element images are arranged in a delta array. Each element image as a circular, when the size of the diameter is R, the vertical pitch of the elemental images of the delta arrangement (the vertical direction of the lens pitch) is {R · (SRQT (3) / 2)} be. When the vertical lens pitch in the figure (a) corresponds to the A pixel and the vertical lens pitch in the figure (b) corresponds to the B pixel, the positions of the pixels newly added by the size conversion. Is expressed using the following equation (1). Here, B> A.

The floor function is used on both sides of the inequality in equation (1). The floor function is a function that takes one actual argument and returns the maximum integer that does not exceed the actual argument.
The pixel position to which a new pixel is added at the time of size conversion is the value of n when the above equation (1) is satisfied.

Although the case of converting the size in the vertical direction is specifically described here, the same concept can be applied to the size conversion in the horizontal direction.

FIG. 16 is a schematic view showing a reference relationship of pixel values in the size conversion process by the image size conversion unit. In addition, this figure shows the relationship of the pixel value reference before and after the size conversion by the cross-sectional view of the pixel array. The figure shows a cross section of only a part of the element image group. In the figure, each of the broken lines H1, H2, and H3 indicates the boundary of the element image. That is, in the cross section shown, the range from the broken lines H1 to H2 corresponds to one element image. Further, the range from the broken line H2 to H3 corresponds to another element image. The distance from the center of the element image to the center of another adjacent element image is the lens pitch (pitch of the element image). The distance from the broken line H1 to H2 and the distance from the broken line H2 to H3 are also equal to the lens pitch, respectively. Then, in the illustrated example, the lens pitch before the size conversion corresponds to 5.5 pixels (value of A). Further, the lens pitch after the size conversion corresponds to 6.0 pixels (value of B). That is, after the size conversion, the lens pitch is equal to an integer number of pixels. That is, in this example, while there are pixels belonging to a plurality of element images before the size conversion, a certain pixel always belongs to only one element image after the size conversion. In this example, A = 5.5 and B = 6.0 are set for easy understanding. Actually, the one-element image may be configured as an array of tens or hundreds of pixels in each of the vertical and horizontal directions.

In the figure, for convenience, each pixel is numbered.
Before the size conversion, the one-element image corresponds to 5.5 pixels. The first element image (range from broken lines H1 to H2) includes pixels 1 to 5, and also includes the upper half of pixel 6. The second element image (range from broken line H2 to H3) includes the lower half of the sixth pixel and also includes the seventh to eleventh pixels.
After the size conversion, the one-element image corresponds to 6.0 pixels. The first element image (range from broken line H1 to H2) includes pixels 1 to 6. Further, the second element image (range from broken line H2 to H3) includes pixels 7 to 12.

The image size conversion unit refers to the pixel value of the image before resizing and generates the image after resizing. In the illustrated example, the 12th pixel in the resized image is a newly added pixel. When the size is changed in the direction of increasing the number of pixels, the image size conversion unit always arranges the added pixels at the positions of the ends of the lens (edges of the element images). In the image after the size conversion, the pixels 1 to 11 each have a pixel value referring to the pixels 1 to 11 before the size conversion. That is, the image size conversion unit sets the pixel value of the first pixel before conversion as the pixel value of the first pixel after conversion, and sets the pixel value of the second pixel before conversion as the pixel value of the second pixel after conversion. The pixel value is the same as below. Then, the 12th pixel in the image after the size conversion has a pixel value referring to the 11th pixel in the image before the size conversion.

In this way, since the positions of the added pixels can be obtained by calculation, the image before resizing can be completely restored based on the image after resizing. That is, in this modification, the process of size conversion in the direction of increasing the number of pixels is reversible conversion.

FIG. 17 is a schematic view showing a multi-viewpoint image group generated based on the element image group that has been resized according to the present modification. In the illustrated example, the multi-viewpoint image group includes a multi-viewpoint image from a total of 64 viewpoints, 8 in the vertical direction and 8 in the horizontal direction. The element image group that is the basis of the multi-viewpoint image group shown is the one that has undergone size conversion processing in both the vertical direction and the horizontal direction. That is, pixels are added in both the vertical direction and the horizontal direction by the size conversion process. Then, the multi-viewpoint image group including the pixels added by the size conversion process is shown by enclosing it in a frame in the figure. That is, among the multi-viewpoint image group arranged in 8 rows × 8 columns, the pixels added by the size conversion process are added to the images of all the viewpoints in the 8th row and the images of all the viewpoints in the 8th column. included. Even if the compression ratio of the images at these viewpoints is increased (that is, the size of the code is reduced) during coding, the added pixels are also deleted on the decoding side, resulting in deterioration of image quality. It can be suppressed.

That is, according to this modification 3, the following effect can be obtained.
The first effect is that in the process of converting the image size, the pixel value of the pixel before conversion is used as it is as the pixel value of the pixel after conversion, so that the amount of calculation processing in the size conversion process can be reduced. That is, it is not necessary to perform arithmetic processing for interpolation of pixel values.
The second effect is that the image size conversion can be a reversible conversion.
The third effect is that the deterioration of the image quality of the pixels added during the size conversion process does not result in the final code deterioration. This is because the pixels added during the size conversion process on the coding side are deleted during the size conversion process on the decoding side.

Further, the information of the multi-viewpoint image including the pixels added during the size conversion process on the coding side may not be included in the code. This can be done, for example, by preventing the multi-viewpoint image selection unit 54 shown in FIG. 9 from selecting the multi-viewpoint image including the added pixels.

The functions of the element image group generating device, the coding device, the decoding device, and the mask image generating device in each of the above-described embodiments and modifications thereof may be realized by a computer.
In that case, a program for realizing the functions of each of these devices may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. .. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

The present invention can be used in businesses that generate content using stereoscopic images (videos), such as broadcasting businesses and content distribution businesses.

1, 2, element image group generation device 3 coding device 4 decoding device 10, 11, 13, 14 Mask image storage unit 21 Multi-viewpoint image group acquisition unit 22 Image size conversion unit 23 Element image external component subtraction unit 31 Image conversion unit 32 Image size conversion unit 33 Element image group output unit 40 Mask image generation device 41 Three-dimensional model storage unit 42 Element image group generation unit 43 Image size conversion unit 44 Image conversion unit 45 Inversion unit 46 Mask image output unit 51 Element image group acquisition unit 52 Image size conversion unit 53 Image conversion unit 54 Multi-viewpoint image selection unit 55 Multi-viewpoint image coding unit 61 Multi-viewpoint image decoding unit 62 Insufficient multi-viewpoint image generation unit 63 Element external component subtraction unit 64 Image conversion unit 65 Image size conversion Part 66 Element image group output part

Claims (4)

An element image group acquisition unit that acquires an element image group consisting of element images,
An image conversion unit that converts the element image group acquired by the element image group acquisition unit into a multi-viewpoint image group, and an image conversion unit.
A mask image storage unit that stores a mask image that holds the degree to which components other than the element image are included in the pixels in the multi-viewpoint image group as pixel values of the pixels.
The image conversion unit by referring to a pixel value indicating the degree to which components other than the element image of the mask image stored in the mask image storage unit are included in the pixels in the multi-viewpoint image group. For the image of each viewpoint included in the multi-viewpoint image group output from, the average value of the pixel values of the area of the mask image corresponding to the area of the image of the viewpoint is calculated, and the average value is determined by the component outside the element image. A multi-viewpoint image selection unit that determines whether or not it represents a value or less, and selects and outputs only an image of a viewpoint indicating that the component outside the element image is equal to or less than the predetermined value.
A multi-viewpoint image coding unit that encodes and outputs only the image of the viewpoint selected by the multi-viewpoint image selection unit in the multi-viewpoint image group, and
A coding device comprising. For each of the element images included in the element image group acquired by the element image group acquisition unit, at least one of the vertical direction and the horizontal direction so that the pitch of the element image is an integral multiple of the pixel pitch. Image size converter, which converts image size by inserting or deleting pixels in
Further equipped,
The image conversion unit converts the element image group whose size has been converted by the image size conversion unit into the multi-viewpoint image group.
The coding apparatus according to claim 1. When converting the image size, the image size conversion unit sets the pixel value of the pixel included in the element image before conversion as the pixel value of the pixel at the corresponding position included in the element image after conversion.
The coding apparatus according to claim 2. Computer,
An element image group acquisition unit that acquires an element image group consisting of element images,
An image conversion unit that converts the element image group acquired by the element image group acquisition unit into a multi-viewpoint image group, and an image conversion unit.
A mask image storage unit that stores a mask image that holds the degree to which components other than the element image are included in the pixels in the multi-viewpoint image group as pixel values of the pixels.
The image conversion unit by referring to a pixel value indicating the degree to which components other than the element image of the mask image stored in the mask image storage unit are included in the pixels in the multi-viewpoint image group. For the image of each viewpoint included in the multi-viewpoint image group output from, the average value of the pixel values of the area of the mask image corresponding to the area of the image of the viewpoint is calculated, and the average value is determined by the component outside the element image. A multi-viewpoint image selection unit that determines whether or not it represents a value or less, and selects and outputs only an image of a viewpoint indicating that the component outside the element image is equal to or less than the predetermined value.
A multi-viewpoint image coding unit that encodes and outputs only the image of the viewpoint selected by the multi-viewpoint image selection unit in the multi-viewpoint image group, and
A program for functioning as an encoding device.

Images