JP6979290B2 - Image coding device and image decoding device, as well as image coding program and image decoding program. - Google Patents

Description

The present invention relates to an image coding device and an image decoding device, and an image coding program and an image decoding program for a stereoscopic image using an integral method.

Conventionally, as one of the imaging / display methods for a stereoscopic television that can be visually recognized from an arbitrary viewpoint, an integral method using a lens group (lens array) arranged on a plane or a spherical surface has been used. The output signal from the camera using this integral method is an integrated stereoscopic imaging signal obtained from the lens group.
In the shooting of a stereoscopic image by the integral method based on the basic principle, as shown in FIG. 15, the subject 1 is photographed by the image pickup camera 4 through the lens 2 and the lens array 3 in which a large number of minute lenses are arranged. When displaying a stereoscopic image, an element image group in which images taken by each lens (hereinafter referred to as "element image") are arranged is displayed on a display device 5 such as a liquid crystal panel, and is displayed in front of the element image group. By arranging the lens array 6, the stereoscopic image 7 is reproduced in space by reproducing the light rays emitted by the original subject 1.

It is known that the resolution of the reproduced stereoscopic image is related to the number of an image pickup camera or an image pickup element and the number of lenses (hereinafter, referred to as "element lenses") constituting the above lens group (lens array). There is. As the number of element lenses increases and the number of pixels of the pixels (hereinafter referred to as "element pixels") assigned to one lens increases, the resolution of the reproduced stereoscopic image also increases.

Considering the transmission and recording of an element image group composed of element images capable of reproducing a natural (that is, an image quality that can be recognized without any problem that the image is three-dimensional), the data size is determined after encoding. However, it increases because the screen resolution is high.

In AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), which are conventional coding methods used for data size compression, a direct conversion process such as a discrete cosine transform is performed on a coded image. In this process, the frequency components included in the image are separated for each frequency. Since it is difficult for the human eye to notice a change in the pattern of a high frequency component, in the coding process of AVC or HEVC, a process of reducing the data size by quantizing the high frequency component is performed.
Here, a case of encoding an element image group as shown in FIG. 16A will be considered. Note that FIG. 16B is a diagram showing each element image by enlarging a part of the element image group. Each pixel included in the element image indicates the luminance information of a light ray coming from a specific direction according to the pixel position.

In the coding of the element image group as shown in FIG. 16A, when the screen resolution of the element image and the image size for which the orthogonal conversion is performed in the coding process do not match, one element image is affected by the adjacent element image. Therefore, many high-frequency components are present in the image after orthogonal conversion. Since these high-frequency components are rounded by the quantization process, the more the rounding is, the more the image deterioration at the time of coding becomes remarkable.
As a method for solving this problem, a technique for matching the size of an element image and an image to be subjected to a orthogonal transformation process has been developed (see, for example, Patent Document 1).

Further, as a technique for efficiently encoding an element image group, the technique described in Patent Document 2 has been developed. In the technique described in Patent Document 2, the same pixel position is obtained from all element images based on the fact that each pixel constituting the element image represents the intensity of light rays coming from a specific direction according to the pixel position. By extracting and collecting the pixels in the above, an image visually recognized from a specific direction (viewpoint) (hereinafter referred to as "multi-viewpoint image") is generated. That is, the element image group is converted into a set having pixels at the same position in each element image as elements, and an image group in which multi-viewpoint images are collected (hereinafter referred to as "multi-viewpoint image group") is generated. The multi-viewpoint image group is a group of multi-viewpoint images arranged in the horizontal direction and the vertical direction according to the respective viewpoint positions. Then, a method of performing coding processing on a multi-viewpoint image group in which these multi-viewpoint images are collected has been reported. FIG. 17 shows an example of a multi-viewpoint image group converted from an element image group. In FIG. 17, the positional relationship between the character in the foreground and the fish in the background is different depending on each viewpoint.

As for the method of encoding after converting to the multi-viewpoint image group, the method of coding all the multi-viewpoint images and the depth image information (hereinafter referred to as "depth") from the multi-viewpoint image group are used. A method of encoding a part of a multi-viewpoint image and the generated depth by generating it has been reported.

A method for encoding a part of the multi-viewpoint image and the generated depth will be described with reference to FIG.
In FIG. 18, among the multi-viewpoint images converted from the element image group, the multi-viewpoint image corresponding to the edges of the upper, lower, left, and right visual areas and the multi-viewpoint image at the center of the visual range are extracted, and the extracted multi-viewpoint image is obtained. And the depth corresponding to those images are the targets of encoding. On the encoding side, since the number of images encoded for transmission or recording is smaller than that in the case of encoding all images, efficient data compression can be performed. On the other hand, on the decoding side, a process of interpolating (hereinafter, referred to as “viewpoint interpolation”) a multi-viewpoint image that has not been transmitted is required. In viewpoint interpolation, it is possible to generate an image close to the original image by using the depth generated on the encoding side.

Japanese Unexamined Patent Publication No. 2006-148885 Japanese Unexamined Patent Publication No. 2013-251663

When the above-mentioned method of encoding a part of the multi-viewpoint image and the generated depth is used, the correlation between the images between the viewpoints is high at present, and if the image is generated by CG (Computer Graphics), the decoding side. It is possible to obtain a multi-viewpoint image group with little deterioration in image quality. However, there is room for improvement in the fact that the live-action video cannot generate a video with sufficient image quality that can withstand the practical service on the decoding side.

The present invention has been made in view of the above points, and is an image encoding device capable of restoring a stereoscopic image in which deterioration of image quality is suppressed while performing efficient data compression of a stereoscopic image. An object of the present invention is to provide an image decoding device, and an image coding program and an image decoding program.

In order to solve the above problems, the image coding device of the present invention is an image coding device that captures and encodes a multi-viewpoint image group generated from a plurality of element images obtained via a lens array. The configuration includes a sweeping processing means and an image coding means.

According to such a configuration, the image coding apparatus uses the plurality of element images obtained from the light beam obtained through the plurality of element lenses constituting the lens array by the sweeping processing means, and the same position in each of the plurality of element images. a partial image of the extraction, the extracted partial images arranged according to the arrangement of the element images capture the multi-viewpoint image group generated by a set of partial images, the direction set in advance among the multi-viewpoint image group In, for each predetermined number of multi-viewpoint images, a multi-viewpoint image to be a reference point to be encoded is determined, and for each frame of the captured image, a multi-viewpoint image to be encoded in the next frame is determined. The sweeping process indicating that the multi-viewpoint image to be encoded in the frame is switched to the adjacent multi-viewpoint image in a preset direction is performed so that all the multi-viewpoint images in the multi-viewpoint image group are to be encoded. It can be repeated for a number of frames.
As a result, the image coding device does not always encode the multi-viewpoint image at the same position as in the prior art, but switches the multi-viewpoint image encoded for each frame to the adjacent multi-viewpoint image. Therefore, if a plurality of frames are coded, the multi-viewpoint image can be coded from all viewpoints. That is, it is possible to eliminate the unencoded multi-viewpoint image, and it is possible to make the image quality at the time of decoding uniform by using the multi-viewpoint image of more viewpoints.

Further, the image coding apparatus generates depth image information of the multi-viewpoint image determined by the sweeping processing means as the coding target by the image coding means, and the multi-viewpoint image determined as the coding target and the multi-viewpoint. Encoded data can be generated by compressing and encoding the depth image information generated for the image.
As a result, the image coding apparatus can compress and encode the multi-viewpoint image determined to be the target of coding by the sweeping processing means and the depth image information thereof among all the multi-viewpoint images to generate the coded data. can. Therefore, as compared with the case where all the multi-viewpoint images are encoded, the number of multi-viewpoint images to be encoded can be reduced, so that the data size of the coded data can be reduced.

As described above, the image coding apparatus of the present invention can generate coded data by switching the multi-viewpoint image to be coded for each frame to the adjacent multi-viewpoint image in a preset direction. As a result, compared to the case of capturing the coded data obtained by encoding the multi-viewpoint image at the same position, the image decoding device recovers the missing image by capturing the coded data by the image coding device. In this case, the image quality can be made uniform and the deterioration of the image quality of the stereoscopic image can be suppressed.

Further, the image decoding apparatus of the present invention is configured to include an image decoding means for acquiring the coded data generated by the above-mentioned image coding apparatus and a multi-viewpoint image generating means.

According to such a configuration, the image decoding device acquires the coded data generated by the image coding device by the image decoding means, and uses the coded data as the multi-viewpoint image determined to be the object of coding. It can be decoded into the depth image information generated for the multi-viewpoint image.
Further, the image decoding device uses the multi-viewpoint image generation means to decode the insufficient multi-viewpoint images excluding the decoded multi-viewpoint images among all the multi-viewpoint images for each frame adjacent to each other in the same frame. It is insufficient by extracting the multi-viewpoint image and the decoded multi-viewpoint image adjacent between the frames as the reference multi-viewpoint image and executing the viewpoint insertion process using the reference multi-viewpoint image and its depth image information. Multi-viewpoint images can be interpolated to generate and output multi-viewpoint image groups for each frame.
As a result, the image decoding device generates a multi-viewpoint image group by performing viewpoint interpolation using the multi-viewpoint image between frames in addition to the viewpoint interpolation using the multi-viewpoint image within one frame. Can be done. Therefore, it is possible to suppress the deterioration of the image quality of the multi-viewpoint image to be interpolated.

As described above, the image decoding device of the present invention acquires the coded data generated by the image coding device, performs viewpoint interpolation within one frame or between frames, and generates a missing multi-viewpoint image. can do. Further, since the information of the multi-viewpoint image can be used from more viewpoints by the sweeping process of the image coding device, the image quality of the multi-viewpoint image group can be made uniform and the deterioration of the image quality of the stereoscopic image can be suppressed. Therefore, it is possible to generate a stereoscopic image having sufficient image quality that can withstand practical services even for a live-action image having a high screen resolution and a large data size.

According to the present invention, an image coding device and an image decoding device, an image coding program, and an image coding program capable of restoring a stereoscopic image in which deterioration of image quality is suppressed while performing efficient data compression of a stereoscopic image are performed. An image decoding program can be provided.

It is a figure for demonstrating the concept of the sweeping process executed by the image coding apparatus which concerns on this embodiment. It is a functional block diagram which shows the structure of the image coding apparatus and the image decoding apparatus which concerns on this embodiment. It is a figure which shows the multi-viewpoint image group which represented each multi-viewpoint image and the depth thereof simply. It is a figure which shows the multi-viewpoint image which performs the coding in the multi-viewpoint image group. It is a figure which shows the direction of the sweeping process which switches a multi-viewpoint image to be coded for every frame. It is a figure which shows the example of the multi-viewpoint image which performs the coding in the 1st frame. It is a figure which shows the example of the multi-viewpoint image which performs the coding in the 2nd frame. It is a figure which shows the example of the multi-viewpoint image which performs the coding in the 3rd frame. It is a figure which shows the example of the multi-viewpoint image which performs the coding in the 4th frame. It is a figure which shows the example of the multi-viewpoint image which performs the coding in the 5th frame. It is a figure for demonstrating the multi-viewpoint image referred to in the viewpoint interpolation processing. It is a figure which shows the other example of the multi-viewpoint image which performs the coding in the multi-viewpoint image group. It is a figure of another example which shows the direction of a sweeping process which switches a multi-viewpoint image to be coded frame by frame. It is a functional block diagram which shows the structure of the image coding apparatus and the image decoding apparatus which concerns on this embodiment. It is a figure for demonstrating the technique of taking and displaying a 3D image by an integral method. FIG. 16A is a diagram showing an example of an element image group. FIG. 16B is a diagram showing an enlarged view of the element image. It is a figure which shows an example of the multi-viewpoint image group converted from the element image group. It is a figure for demonstrating the technique of encoding a part of a multi-viewpoint image and its depth.

Hereinafter, embodiments for carrying out the present invention (hereinafter, referred to as “embodiments”) will be described with reference to the drawings.
First, an outline of the processing executed by the image coding device 10 and the image decoding device 20 (both described later, see FIG. 2) according to the present embodiment will be described.

The image coding apparatus 10 according to the present embodiment has an improved method for encoding a part of the above-mentioned multi-viewpoint image and the generated depth, and has the following configuration. (1) Increase the number of multi-viewpoint images to be coded, and perform coding within the range that can be handled by the current viewpoint synthesis technology. That is, as compared with the case of encoding the multi-viewpoint image corresponding to the edges of the upper, lower, left, and right visual ranges and the multi-viewpoint image at the center of the visual range in the conventional technique described with reference to FIG. Among them, the number of multi-viewpoint images to be encoded is increased. (2) A new multi-viewpoint to be encoded by switching the multi-viewpoint image to be encoded for each frame of the moving image to an adjacent multi-viewpoint image (for example, a multi-viewpoint image in the horizontal direction or the vertical direction). Performs a sweeping process to select an image.

The concept of the sweeping process executed by the image coding apparatus 10 according to the present embodiment will be described with reference to FIG.
In the conventional coding of a multi-viewpoint image group, the position of the coded multi-viewpoint image in the entire image is fixed in advance. For example, as shown in FIG. 18, images at the edges of the upper, lower, left, and right visual areas and the center of the visual areas were selected in advance, and the multi-viewpoint images at the fixed positions were encoded in all frames. That is, the multi-viewpoint image to be encoded was fixed both at the position on the spatial axis within one frame and at the position on the time axis between the frames.
On the other hand, in the image coding apparatus 10 of the present embodiment, the multi-viewpoint image to be coded is switched for each frame. For example, as shown in FIG. 1, in the first frame of the moving image, the “multi-viewpoint image 1” and the “multi-viewpoint image 3” are extracted and encoded. In the second frame, the "multi-viewpoint image 2" and the "multi-viewpoint image 4" are extracted and encoded. Similarly, in the third and subsequent frames, the multi-viewpoint image to be encoded is switched for each frame. In this embodiment, the process of switching and determining the multi-viewpoint image to be encoded for each frame is referred to as a sweeping process.

By performing this sweeping process, in the image decoding device 20, in addition to the viewpoint interpolation within one frame space (viewpoint interpolation using an image having a short distance on the spatial axis), an image between frames can be obtained. It is possible to perform the viewpoint interpolation (viewpoint interpolation using an image having a short distance on the time axis) (details will be described later with reference to FIG. 11). Therefore, the image quality of the generated image can be improved. In addition, when a multi-viewpoint image that is always in the same position is interpolated from the viewpoint, the image quality deterioration of the unencoded image (multi-viewpoint image with viewpoint interpolation) is compared with the coded / decoded multi-viewpoint image. It stands out. By performing the sweeping process, an image containing the information of the multi-viewpoint image from all viewpoints is generated, so that the image quality of the multi-viewpoint image group output on the decoding side becomes uniform and changes depending on the position where the stereoscopic image is viewed. It is possible to suppress deterioration of the image quality of the stereoscopic image.

<Image coding device, image decoding device>
Next, the image coding device 10 and the image decoding device 20 according to the present embodiment will be described with reference to FIG.
The image coding device 10 is connected to or incorporated into, for example, an integral type photographing device (imaging camera) to convert element image group data into a multi-viewpoint image group, encode it, and decode the image. It is a device that outputs to the conversion device 20 side. The image decoding device 20 acquires coded data from the image coding device 10 via a transmission device (not shown) or the like, decodes the coded data to generate a multi-viewpoint image group, and generates an original element. It is a device that restores image group data. The image decoding device 20 is connected to or incorporated into, for example, an integral type image display device or a projector device.

≪Image coding device≫
As shown in FIG. 2, the image coding device 10 includes a multi-viewpoint image conversion means 11, a sweeping processing means 12, and an image coding means 13.

The multi-viewpoint image conversion means 11 captures a plurality of element images (element image group data) obtained from a light flux obtained through a plurality of element lenses constituting a lens array (not shown). Then, the multi-viewpoint image conversion means 11 arranges partial images at the same position in each element image of the captured plurality of element images (element image group data) according to an array in the element image (part). Convert to a set of images). The partial image is an image obtained by cutting out a part of the element image, and may be composed of one pixel or a plurality of adjacent pixels.

In the present embodiment, element image group data taken by an integral type stereoscopic television camera is captured, and an element image group when a stereoscopic image is formed on a display device using a lens array composed of square element lenses. Will be described as an example.
Here, the resolution of the element image is set to 20 × 20 pixels as an example. Further, the number of effective lenses in the lens array arranged in the display device is 240 × 160 as an example. In this case, the size of the element image group is 4800 × 3200 pixels. Further, in the multi-viewpoint image group obtained by converting this element image group, the multi-viewpoint image composed of 240 × 160 pixels has 20 × 20 viewpoints.
Hereinafter, as shown in FIG. 3, the multi-viewpoint image and its depth are simplified and represented by one rectangle for explanation. Further, FIG. 3 shows that there is a 20 × 20 viewpoint multi-viewpoint image as a multi-viewpoint image group (reference numeral X in FIG. 3) converted from the element image group.

As shown in FIG. 2, the sweeping processing means 12 performs a process of determining a multi-viewpoint image to be encoded for each multi-viewpoint image (and its depth; hereinafter appropriately omitted) for each frame. Then, the sweeping processing means 12 switches the multi-viewpoint image to be encoded for each frame to a multi-viewpoint image at an adjacent viewpoint position (for example, a horizontal direction or a vertical direction).
Here, for example, the sweeping processing means 12 encodes 100 viewpoints (multi-viewpoint image whose inside of the rectangle is shown by diagonal lines) out of 20 × 20 viewpoints (400 viewpoints in total) of the multi-viewpoint image. It is determined in advance as a reference point for the viewpoint image. Then, the sweeping processing means 12 executes a sweeping process in which the viewpoint position is moved every time the frame is switched in the direction shown in FIG. 5 in which the 100 viewpoints are set in advance. As a result, the sweeping processing means 12 is referred to as a first frame shown in FIG. 6, a second frame shown in FIG. 7, a third frame shown in FIG. 8, a fourth frame shown in FIG. 9, and a fifth frame shown in FIG. In this way, the positions of the multi-viewpoint images to be encoded are switched. In this case, if there are four frames, the components of the multi-viewpoint image (and its depth) of all viewpoints can be included.
Further, since the multi-viewpoint image to be coded has 100 viewpoints as compared with the case where the sweeping processing means 12 encodes all 400 viewpoints, the coded data size can be reduced.

The number of viewpoints for which the sweeping processing means 12 determines coding in one frame and the sweeping direction (horizontal direction, vertical direction, diagonal direction, etc.) switched for each frame are set in advance.

As shown in FIG. 2, the image coding means 13 generates depth (depth image information) for a multi-viewpoint image determined by the sweeping processing means 12 to be encoded. Then, the image coding means 13 compresses and encodes the multi-viewpoint image determined to be encoded and its depth by, for example, a conventional coding method for a two-dimensional image (AVC, HEVC, etc.) to obtain the coded data. Generate and output this coded data. The output coded data is transmitted to the image decoding device 20 by a transmission device or the like (not shown). For the generation and coding of the depth of the multi-viewpoint image, for example, the technique described in JP-A-2007-36800 can be used.

≪Image decoding device≫
As shown in FIG. 2, the image decoding device 20 includes an image decoding means 21, a multi-viewpoint image generation means 22, and an element image conversion means 23.

The image decoding means 21 acquires the coded data generated by the image coding device 10 via, for example, a transmission device, decodes the coded data, and acquires a multi-viewpoint image and its depth.

The multi-viewpoint image generation means 22 interpolates the multi-viewpoint image that was not transmitted on the decoding side by executing the viewpoint interpolation processing using the multi-viewpoint image decoded by the image decoding means 21 and its depth. , Executes the process of generating a multi-viewpoint image group.
The multi-viewpoint image generation means 22 performs this viewpoint interpolation processing on a preset multi-viewpoint image (an image having a close distance in the same frame) on the spatial axis or a multi-viewpoint image adjacent on the time axis. By referring to the viewpoint image (the image in which the distance between frames is short), the missing multi-viewpoint image is generated.
After determining the image to be used when generating the multi-viewpoint image, the multi-viewpoint image generation means 22 uses, for example, the viewpoint synthesis technique for performing viewpoint interpolation described in Japanese Patent Application Laid-Open No. 2010-20188. To generate the missing multi-viewpoint image.
Hereinafter, the process of determining the multi-viewpoint image to be referred to for performing the viewpoint interpolation by the multi-viewpoint image generation means 22 will be described with reference to the examples shown in FIGS. 4 to 10 and the like.

FIG. 11 is a diagram for explaining the viewpoint interpolation processing in the case where the sweeping processing for switching the multi-viewpoint image to be encoded is performed in the order of each frame shown in FIG. In FIG. 11, out of the 20 × 20 viewpoints (400 viewpoints in total) of the multi-viewpoint image, the 5 × 5 viewpoints, which are a part of each frame, are extracted and shown for the first to seventh frames. For example, the first frame shows a multi-viewpoint image included in the region of reference numeral α shown in FIG. In the 1st to 7th frames shown in FIG. 11, it is considered to perform viewpoint interpolation of the viewpoint of the 3rd row and 3rd column located at the center.
Since the multi-viewpoint image to which the number "55" is assigned already exists in the third frame, it is not necessary to perform viewpoint interpolation. Similarly, it is not necessary to perform viewpoint interpolation for the multi-viewpoint image to which the number "54" in the 7th frame is assigned.

Here, in the multi-viewpoint image generation means 22, for example, the multi-viewpoint image to which the number "55" in the third frame is assigned and the multi-viewpoint image to which the number "54" in the seventh frame is assigned are the same. If it is (for example, the same luminance value), there is a high possibility that the 4th to 6th frames are the same image, and the image can be interpolated by copying the image.
It should be noted that instead of comparing the entire multi-viewpoint image to which the number "55" in the third frame is assigned and the entire multi-viewpoint image to which the number "54" in the seventh frame is assigned, the area divided thereof. You can handle it with. If the multi-viewpoint image to which the number "55" is assigned in the 3rd frame and the partial area of the multi-viewpoint image to which the number "54" in the 7th frame is assigned are the same, in the 4th to 6th frames. Since there is a high possibility that an image region having the same positional relationship as the same partial region is also the same image, it is possible to interpolate by copying the image region. Further, as an example of a partial region in this case, it is also possible to use a division block in coding as a unit.

Further, for example, when a parallax occurs between the multi-viewpoint image to which the number "55" in the third frame is assigned and the multi-viewpoint image to which the number "54" in the seventh frame is assigned, on the time axis. It can be considered that the change in is due to the pan and tilt of the camera and the movement of the subject, and the multi-viewpoint image generation means 22 performs the viewpoint insertion process after predicting the movement.

For example, in the viewpoint interpolation of the multi-viewpoint image of the 3rd row and 3rd column of the 2nd frame, in addition to the difference information on the time axis described above, the numbers "51" and "55" whose distances on the spatial axis are close are short. Perform viewpoint interpolation using the multi-viewpoint image assigned to.
Similarly, in the viewpoint interpolation of the multi-viewpoint image in the 3rd row and 3rd column of the 4th frame, in addition to the difference information on the time axis, the numbers "55" and "60" that are close to each other on the space axis are assigned. Viewpoint interpolation is performed using the obtained multi-viewpoint image.
Similarly, in the viewpoint interpolation of the multi-viewpoint image in the 3rd row and 3rd column of the 6th frame, in addition to the difference information on the time axis, the numbers "50" and "54" that are close to each other on the spatial axis are assigned. Viewpoint interpolation is performed using the obtained multi-viewpoint image.

In addition, in the viewpoint insertion of the multi-viewpoint image in the 3rd row and 3rd column of the 1st frame, since there are many multi-viewpoint images that can be referred to in the vertical and horizontal spatial axis directions, for example, the same distance is set both in the vertical and horizontal directions. Viewpoint insertion is performed using certain multi-viewpoint images "46" and "65", "55" and "56".
Similarly, in the viewpoint interpolation of the multi-viewpoint image in the third row and third column of the fifth frame, for example, "45" and "64", "54" and "54", which are multi-viewpoint images at the same distance in both the top, bottom, left and right. 55 ”is used to perform viewpoint interpolation.

Further, for example, as a method of using both a multi-viewpoint image adjacent on the spatial axis (an image having a short distance within the same frame) and a multi-viewpoint image adjacent on the time axis (an image having a short distance between frames). You may also do the following: The multi-viewpoint image generation means 22 has eight multi-viewpoint images adjacent (surrounding) within the same frame of the multi-viewpoint image to be interpolated, and nine multi-viewpoints (18 in total) in each of the previous and next frames. Viewpoint interpolation is performed using information about a multi-viewpoint image at a position corresponding to a cube composed of an image.
For example, in the viewpoint interpolation of a multi-viewpoint image in the third row and third column of the second frame, the multi-viewpoint image generation means 22 uses the multi-viewpoint image (an image having a short distance in the same frame) adjacent to each other on the spatial axis. A multi-viewpoint image to which the numbers "55" and "51" are assigned is extracted. Further, the multi-viewpoint image generation means 22 is assigned the numbers "51" and "61" of the first frame as multi-viewpoint images (images having a short distance between frames) adjacent to each other on the time axis. Is extracted, and a multi-viewpoint image to which the numbers "51" and "60" are assigned in the third frame is extracted.

As described above, the multi-viewpoint image generation means 22 performs viewpoint interpolation using the multi-viewpoint images adjacent on the spatial axis and the multi-viewpoint images adjacent on the time axis, and obtains the unencoded multi-viewpoint image. Interpolate.
It should be noted that, among the multi-viewpoint images adjacent on the spatial axis and the time axis, which multi-viewpoint image is referred to by the multi-viewpoint image generation means 22 during the viewpoint interpolation processing is determined in advance by the multi-viewpoint image generation means. It shall be set to 22.

As shown in FIG. 2, the element image conversion means 23 converts the multi-viewpoint image group generated by the multi-viewpoint image generation means 22 by viewpoint insertion into a plurality of element images (element image groups). Then, it is restored and output as element image group data.
Specifically, the element image conversion means 23 extracts pixels at the same position from each partial image in the multi-viewpoint image group generated by the multi-viewpoint image generation means 22, and these extracted pixels are used in the element lens. The element image group is restored by making the element image arranged according to the position of the partial image indicated by the lens coordinates.

As described above, according to the image coding device 10 and the image decoding device 20 according to the present embodiment, the image coding device 10 performs a sweeping process to obtain a multi-viewpoint image to be coded frame by frame. , It is possible to switch to an adjacent multi-viewpoint image in a preset direction, determine the image, and encode the image. Then, the image decoding device 20 performs viewpoint interpolation within one frame space (on the space axis) and viewpoint interpolation between frames (on the time axis) to generate a missing multi-viewpoint image. Can be done. Therefore, it is possible to generate an image containing information on the multi-viewpoint image from more viewpoints by performing the sweeping process, as compared with the case where the multi-viewpoint image that is always at the same position is inserted into the viewpoint as in the prior art. .. Therefore, the image quality of the multi-viewpoint image group output on the decoding side becomes uniform, and the deterioration of the image quality of the stereoscopic image can be suppressed.
Therefore, according to the image coding device 10 and the image decoding device 20, it is possible to restore a stereoscopic image in which deterioration of image quality is suppressed while performing efficient data compression of the stereoscopic image. This makes it possible to generate a stereoscopic image with sufficient image quality that can withstand practical services even for a live-action image having a high screen resolution and a large data size.

The direction for switching the multi-viewpoint image in the sweeping process executed by the sweeping processing means 12 of the image coding apparatus 10 according to the present embodiment is the direction shown in FIG. 5 and the reference point of the multi-viewpoint image shown in FIG. Not limited.
FIG. 12 is another example of the reference point of the multi-viewpoint image, and FIG. 13 is a diagram showing the direction of the sweeping process from the reference point of FIG. The sweeping processing means 12 is determined in advance as a reference point of the multi-viewpoint image for encoding the 100 viewpoints (multi-viewpoint image whose inside of the rectangle is indicated by diagonal lines) shown in FIG. Then, the sweeping processing means 12 executes a sweeping process in which the viewpoint position is moved every time the frame is switched in the direction shown in FIG. 13 in which the 100 viewpoints are set in advance. By doing so, the image compounding device 20 can perform viewpoint interpolation including multi-viewpoint images (and their depths) of all viewpoints if there are four frames.

Further, the image coding device 10 and the image decoding device 20 according to the present embodiment may be configured as the image coding device 10a and the image decoding device 20a shown in FIG.
The image coding device 10a includes the sweeping processing means 12 and the image coding means 13, but does not include the multi-viewpoint image conversion means 11 as compared with the image coding device 10 shown in FIG. In this case, the sweeping processing means 12 of the image coding apparatus 10a captures the data of the multi-viewpoint image group as a result of converting the plurality of element images obtained via the lens array.
Similar to the image coding device 10 of FIG. 2, the image coding device 10a switches the multi-viewpoint image to be coded for each frame to the adjacent multi-viewpoint image in a preset direction and converts the coded data. Can be generated. Therefore, as compared with the case of capturing the coded data obtained by encoding the multi-viewpoint image at the same position as in the prior art, the image decoding device 20a captures the coded data by the image coding device 10a. The image quality can be made uniform when restoring the missing image, and the deterioration of the image quality of the stereoscopic image can be suppressed.

The image decoding device 20a includes the image decoding means 21 and the multi-viewpoint image generation means 22, but does not include the element image conversion means 23 as compared with the image decoding device 20 shown in FIG. In this case, the multi-viewpoint image generation means 22 of the image decoding device 20a outputs the generated multi-viewpoint image group.
Similar to the image decoding device 20 of FIG. 2, the image decoding device 20a acquires the coded data generated by the image coding device 10a, performs viewpoint interpolation within one frame or between frames, and is insufficient. It is possible to generate a multi-viewpoint image. At that time, since the information of the multi-viewpoint image can be used from more viewpoints by the sweeping process of the image coding apparatus 10a, the image quality of the multi-viewpoint image group can be made uniform and the deterioration of the image quality of the stereoscopic image can be suppressed.

In the present embodiment, the image coding device and the image decoding device according to the present invention have been described as independent devices, but the present invention is not limited thereto. For example, in the present invention, a general computer hardware resource can be realized by a program (image coding program and image decoding program) that operates as each means of an image coding device and an image decoding device. .. Further, this program (image coding program and image decoding program) can be distributed via a communication line, or can be recorded and distributed on a recording medium such as a CD-ROM.

10, 10a image coding device 11 multi-viewpoint image conversion means 12 sweeping processing means 13 image coding means 20, 20a image decoding device 21 image decoding means 22 multi-viewpoint image generation means 23 element image conversion means

Claims (6)

Using a plurality of element images obtained from light beams obtained through the plurality of element lenses constituting the lens array, a partial image at the same position in each of the plurality of element images is extracted, and the extracted partial image is used as the element. The multi-viewpoint image group generated by arranging according to the arrangement of the images to form a set of the partial images is captured.
A multi-viewpoint image to be a reference point to be encoded is determined for each predetermined number of multi-viewpoint images in a preset direction in the multi-viewpoint image group, and each frame of the captured image is set in the next frame. The sweeping process indicating that the multi-viewpoint image to be encoded is switched from the multi-viewpoint image to be coded in the frame to the adjacent multi-viewpoint image in the preset direction is performed on the multi -viewpoint image group. A sweeping processing means that repeats for the number of frames so that all multi-viewpoint images are subject to encoding,
The sweeping processing means generates depth image information of a multi-viewpoint image determined to be a coding target, and compresses the multi-viewpoint image determined to be a coding target and the depth image information generated for the multi-viewpoint image. An image coding means that converts and generates coded data,
An image coding apparatus comprising: The coded data generated by the image coding apparatus according to claim 1 is acquired, and the coded data is used as the multi-viewpoint image determined to be the target of the coding and the depth image information generated for the multi-viewpoint image. Image decoding means to decode and
Of all the multi-viewpoint images for each frame, the missing multi-viewpoint images excluding the decoded multi-viewpoint images were decoded and the decoded multi-viewpoint images adjacent to each other in the same frame and the decoded multi-viewpoint images adjacent to each other. The multi-viewpoint image is extracted as a reference multi-viewpoint image, and the lacking multi-viewpoint image is interpolated by executing the viewpoint insertion process using the extracted multi-viewpoint image and the depth image information thereof. A multi-viewpoint image generation means that generates and outputs a multi-viewpoint image group for each frame,
An image decoding device characterized by comprising. A plurality of element images obtained from a light beam obtained through a plurality of element lenses constituting the lens array are captured, a partial image at the same position in each of the plurality of element images is extracted, and the extracted partial image is used as the element image. A multi-viewpoint image conversion means for converting the plurality of element images into a multi-viewpoint image group showing the set of the partial images by forming a set of the partial images arranged according to the arrangement of the above.
Among the converted multi-viewpoint images, a multi-viewpoint image to be a reference point to be encoded is determined for each predetermined number of multi-viewpoint images in a preset direction, and next for each frame of the captured image. the multi-view image to be encoded in a frame, the sweeping process indicating that the switch from the multi-viewpoint images that targets encoded in the frame in the multi-viewpoint image adjacent in a direction the preset said multi-viewpoint image A sweeping processing means that repeats for the number of frames so that all multi-viewpoint images in the group are subject to encoding,
The sweeping processing means generates depth image information of a multi-viewpoint image determined to be a coding target, and compresses the multi-viewpoint image determined to be a coding target and the depth image information generated for the multi-viewpoint image. An image coding means that converts and generates coded data,
An image coding apparatus comprising: The coded data generated by the image coding apparatus according to claim 3 is acquired, and the coded data is used as the multi-viewpoint image determined to be the target of the coding and the depth image information generated for the multi-viewpoint image. Image decoding means to decode and
Of all the multi-viewpoint images for each frame, the missing multi-viewpoint images excluding the decoded multi-viewpoint images were decoded and the decoded multi-viewpoint images adjacent to each other in the same frame and the decoded multi-viewpoint images adjacent to each other. The multi-viewpoint image is extracted as a reference multi-viewpoint image, and the lacking multi-viewpoint image is interpolated by executing the viewpoint insertion process using the extracted multi-viewpoint image and the depth image information thereof. A multi-viewpoint image generation means that generates a multi-viewpoint image group for each frame,
Element image conversion means for converting the generated multi-viewpoint image group into a plurality of element images by returning the partial image represented by the multi-viewpoint image group generated by the multi-viewpoint image generation means to the original position on the element lens. When,
An image decoding device characterized by comprising. An image coding program for operating a computer as the image coding device according to claim 1. An image decoding program for operating a computer as the image decoding device according to claim 2.

Images