JP2016127372A - Video encoder, video decoder, video processing system, video encoding method, video decoding method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the coding efficiency of a video of a plurality of view points of different perspective.SOLUTION: A video encoder 1 includes a first encoder 10, a second encoder 20, and an inter-view processing unit 30. The first encoder 10 encodes the original image SIG1 of expansion point of view, and outputs a bit stream SIG6 of expansion point of view. The second encoder 20 encodes the original image SIG2 of reference point of view, and outputs a bit stream SIG7 of reference point of view. The inter-view processing unit 30 performs projective transformation and illumination compensation of a local decoded image SIG3 depending on the relationship of reference point of view and expansion point of view, and generates a local decoded image SIG4 after conversion of the reference point of view, so that the first encoder 10 can use this local decoded image SIG4 after conversion of the reference point of view as a reference image, when performing inter-prediction.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a moving image encoding device, a moving image decoding device, a moving image processing system, a moving image encoding method, a moving image decoding method, and a program.

  2. Description of the Related Art Conventionally, a video expression method called free navigation has been proposed that enables viewing of video from various viewpoints. In video games, free navigation is already generally used, but in recent years, techniques for realizing free navigation using live-action video as material video have been studied.

  As a scene for realizing free navigation using a live-action video as a material video, for example, a video of a soccer game can be distributed to viewers. In this case, a plurality of cameras are fixedly arranged so as to surround the soccer field, the entire soccer field is photographed by the plurality of cameras, and the photographed video is distributed to the viewer as a material video. According to this, when the viewer appropriately operates the viewpoint to be viewed, the video from the operated viewpoint can be created from the plurality of videos distributed as the material video and presented to the viewer.

  As described above, in order to realize free navigation using live-action video as video material, it is necessary to deliver multiple videos (multi-view video) taken from multiple perspectives with different perspectives to the viewer as material video . For this reason, it is preferable to efficiently compress the material image.

  Thus, a multi-view extension method called MV-HEVC has been standardized as an extension method of HEVC (see, for example, Non-Patent Document 1). In MV-HEVC, videos from a plurality of viewpoints with different perspectives can be referred to each other and encoded simultaneously.

  A case where video of two viewpoints is encoded and decoded by MV-HEVC will be described below. In the following, one of the two viewpoints is referred to as a reference viewpoint, and the other is referred to as an extended viewpoint. The reference viewpoint video is encoded by HEVC in the same manner as in the prior art, and the encoded reference viewpoint video is decoded by HEVC in the same manner as in the prior art. On the other hand, for the extended viewpoint video, the local decoded image of the base viewpoint is encoded as the inter prediction reference image, and for the encoded extended viewpoint video, the base viewpoint decoded image is used as the inter prediction reference image. To decrypt. For this reason, when the correlation between the video of the reference viewpoint and the video of the extended viewpoint is high, the inter prediction is performed with reference to the image of the reference viewpoint, so that the highly efficient encoding is performed as compared with the case of performing the intraframe prediction. be able to.

  FIG. 10 is a diagram illustrating an image used as a reference image for inter prediction when encoding and decoding a base viewpoint video and an extended viewpoint video by MV-HEVC. Images P1, P2, P3, and P4 indicate images for each frame at the extended viewpoint, and images P5, P6, P7, and P8 indicate images for each frame at the reference viewpoint. For example, when the base viewpoint image P6 is obtained by inter prediction, the base viewpoint image P5 is used as an inter prediction reference image. When the extended viewpoint image P3 is obtained by inter prediction, not only the extended viewpoint image P2 but also the base viewpoint image P7 is used as a reference image for inter prediction.

ITU-T H.265 High Efficiency Video Coding.

JP 2012-80151 A

  Here, the multi-view video has a feature that the correlation between moving images for each viewpoint is low. However, in the MV-HEVC disclosed in Non-Patent Document 1, as described above, the video of the extended viewpoint is encoded using the local decoded image of the reference viewpoint as the reference image for inter prediction, and the decoded image of the reference viewpoint is used. Is used as a reference image for inter prediction, and the encoded video of the extended viewpoint is decoded. For this reason, in MV-HEVC shown in the nonpatent literature 1, since an image with a low correlation will be used as a reference image, there exists a possibility that encoding efficiency may fall.

  Therefore, it is conceivable to perform geometric transformation as shown in Patent Document 1 on the local decoded image at the base viewpoint and the decoded image at the base viewpoint, and use the result as a reference image for inter prediction.

  However, the geometric transformation shown in Patent Document 1 is performed for each block. For this reason, if it is going to apply the geometric transformation shown in patent documents 1 to MV-HEVC shown in nonpatent literature 1, changes below a video coding layer and a video decoding layer are needed. In addition, since the memory required for the geometric transformation may vary from block to block, there is a possibility that the required amount of bandwidth considering the memory access granularity increases. According to the above, when the geometric transformation shown in Patent Document 1 is applied to MV-HEVC shown in Non-Patent Document 1, the mounting cost increases.

  Therefore, the present invention has been made in view of the above-described problems, and an object of the present invention is to improve the encoding efficiency of videos from a plurality of viewpoints with different perspectives while suppressing mounting costs.

The present invention proposes the following matters in order to solve the above problems.
(1) The present invention is a moving image encoding device (e.g., corresponding to the moving image encoding device 1 in FIG. 1) that generates encoded data by encoding moving images of a plurality of viewpoints with different perspectives. Single-viewpoint encoding means (for example, corresponding to the first encoding unit 10 and the second encoding unit 20 in FIG. 2) for encoding each of the moving images of the plurality of viewpoints for each viewpoint; One of the viewpoints is set as a reference viewpoint, and one of the plurality of viewpoints excluding the reference viewpoint is set as an extended viewpoint, and the moving image of the reference viewpoint is encoded by the single-viewpoint encoding unit. The local decoded image of the reference viewpoint obtained at this time (e.g., corresponding to the local decoded image SIG3 of the reference viewpoint in FIG. 2) is geometrically transformed according to the relationship between the reference viewpoint and the extended viewpoint, and the reference Local decoded image after viewpoint conversion (example) For example, the inter-viewpoint processing unit (equivalent to the inter-viewpoint processing unit 30 in FIG. 2) that generates the local decoded image SIG4 after conversion of the reference viewpoint in FIG. 2 and the inter-viewpoint processing unit Reference image list addition means (for example, the viewpoint in FIG. 2) that enables the local decoded image after the conversion of the base viewpoint to be used as a reference image when the extended viewpoint moving image is encoded by the single-viewpoint encoding means. A video encoding device characterized in that it is equivalent to the inter-processing unit 30).

  According to the present invention, each of the moving images of the plurality of viewpoints is encoded for each viewpoint by the single viewpoint encoding means. Further, the inter-viewpoint processing means geometrically transforms the local decoded image of the reference viewpoint obtained when the moving image of the reference viewpoint is encoded by the single-viewpoint encoding means according to the relationship between the reference viewpoint and the extended viewpoint. Thus, the local decoded image after the conversion of the reference viewpoint is generated. In addition, the reference image list adding means can use the local decoded image after the conversion of the standard viewpoint generated by the inter-viewpoint processing means as a reference image when the extended viewpoint moving image is encoded by the single-viewpoint encoding means. I decided to make it.

  For this reason, a frame of another layer is added to the reference image list, and a moving image can be encoded using a framework similar to MV-HEVC disclosed in Non-Patent Document 1. The change below the encoding layer is not necessary, and the existing HEVC codec design can be utilized to the maximum extent. Therefore, the mounting cost can be suppressed.

  Further, the local decoded image of the reference viewpoint is geometrically transformed according to the relationship between the reference viewpoint and the extended viewpoint to generate a local decoded image after conversion of the reference viewpoint, and the local decoded image after conversion of the reference viewpoint is used. Thus, an extended viewpoint video can be encoded. Here, a multi-view video obtained by photographing the same subject has a feature that a moving image for each viewpoint can be converted into a similar moving image by geometric conversion. For this reason, since a similar image can be used as a reference image by geometric transformation according to the relationship between the base viewpoint and the expanded viewpoint, the encoding performance can be improved.

  (2) The present invention relates to the moving picture coding apparatus according to (1), wherein the inter-viewpoint processing means uses parameters used for geometric transformation (for example, corresponding to each element of a projective transformation matrix s described later), A moving picture coding apparatus is proposed that transmits the coded data to a moving picture decoding apparatus that decodes the coded data.

  According to the present invention, in the moving picture encoding apparatus of (1), the parameters used in the geometric transformation are transmitted to the moving picture decoding apparatus by the inter-viewpoint processing means. For this reason, the moving image decoding apparatus can perform geometric transformation using the parameters used in the moving image encoding apparatus when decoding the encoded data.

  (3) The present invention relates to the moving picture coding apparatus according to (1), wherein the inter-viewpoint processing means is configured to obtain the reference viewpoint obtained when the moving picture at the reference viewpoint is coded by the single-viewpoint coding means. The local decoded image is subjected to geometric transformation and illuminance compensation in accordance with the relationship between the reference viewpoint and the extended viewpoint, and a local decoded image after conversion of the reference viewpoint is generated. Has proposed.

  According to the present invention, in the moving image encoding apparatus of (1), the local decoded image of the reference viewpoint obtained when the moving image of the reference viewpoint is encoded by the single-viewpoint encoding means by the inter-viewpoint processing means. The local decoded image after the conversion of the reference viewpoint is generated by performing geometric conversion and illumination compensation according to the relationship between the reference viewpoint and the extended viewpoint. For this reason, even if the illuminance differs between the moving image of the reference viewpoint and the moving image of the extended viewpoint due to the difference in the shooting environment such as the illumination condition between the reference viewpoint and the extended viewpoint, the difference in illuminance Can be compensated. Therefore, even when the illuminance differs between the reference viewpoint moving image and the extended viewpoint moving image, the encoding performance can be improved by illuminance compensation in accordance with the relationship between the reference viewpoint and the extended viewpoint.

  (4) The present invention relates to the moving picture encoding apparatus according to (3), wherein the inter-viewpoint processing means includes parameters used for geometric transformation (for example, corresponding to each element of a projective transformation matrix s described later), A moving picture coding apparatus that transmits parameters used for illuminance compensation (e.g., corresponding to illuminance compensation parameters a and b described later) to a moving picture decoding apparatus that decodes the encoded data. is suggesting.

  According to the present invention, in the moving picture encoding apparatus of (3), the parameters used in the geometric transformation and the parameters used in the illumination compensation are transmitted to the moving picture decoding apparatus by the inter-viewpoint processing means. It was decided to. Therefore, the moving image decoding apparatus can perform geometric conversion and illuminance compensation using the geometric conversion parameters and the illuminance compensation parameters used in the moving image encoding apparatus when decoding the encoded data.

  (5) In the moving image encoding apparatus according to any one of (1) to (4), the inter-viewpoint processing unit may perform any one of projective transformation and affine transformation by triangular patch division as geometric transformation. We have proposed a video encoding apparatus characterized by performing such a process.

  According to the present invention, in the moving picture encoding apparatus according to any one of (1) to (4), either the projective transformation or the affine transformation by triangular patch division is performed as the geometric transformation by the inter-viewpoint processing means. It can be carried out.

  (6) In the moving image encoding device according to any one of (1) to (5), the reference image list adding unit is configured to convert the reference viewpoint generated by the inter-viewpoint processing unit after conversion of the reference viewpoint. There has been proposed a moving picture coding apparatus characterized in that a decoded picture can be used as a long-term reference frame when the moving picture of the extended viewpoint is coded by the single-view coding means.

  According to the present invention, in the moving picture encoding apparatus according to any one of (1) to (5), the reference decoded image generated by the inter-viewpoint processing unit is converted by the reference image list adding unit, The extended viewpoint video is made available as a long-term reference frame when encoded by single-viewpoint encoding means. This eliminates the need to recalculate the motion vector at the reference viewpoint, thereby reducing the amount of encoding processing.

  (7) The present invention is a video decoding device (for example, equivalent to the video decoding device 100 in FIG. 1) that decodes encoded data obtained by encoding video from a plurality of viewpoints with different perspectives. Then, the encoded data is decoded to generate decoded images of the plurality of viewpoints (for example, the decoded images SIG103 of the extended viewpoint and the decoded images SIG101 and SIG104 of the reference viewpoint in FIG. 5). In addition, single-viewpoint decoding means (for example, the first view of FIG. 5) for deriving parameters (for example, corresponding to each element of a projective transformation matrix s to be described later) used in the geometric transformation at the time of generating the encoded data. And the viewpoint that was the reference viewpoint at the time of generating the encoded data among the plurality of viewpoints as a decoding-side reference viewpoint, and the code If the viewpoint that was the extended viewpoint at the time of data generation is the decoding-side extended viewpoint, the decoded image of the decoding-side reference viewpoint generated by the single-view decoding means (for example, corresponding to the decoded image SIG101 of the reference viewpoint in FIG. 5) ) Is geometrically transformed using the parameters derived by the single-viewpoint decoding means, and the decoded image after conversion of the decoding-side reference viewpoint (for example, the decoded image SIG102 after conversion of the reference viewpoint in FIG. 5) ) For generating the inter-viewpoint processing means (e.g., corresponding to the inter-viewpoint processing unit 130 in FIG. 5) and the decoded side-converted decoded image generated by the inter-viewpoint processing means as the decoding side extension A reference image list adding unit (e.g., corresponding to the inter-view processing unit 130 in FIG. 5) that can be used as a reference image when decoding encoded viewpoint data by the single-viewpoint decoding unit; A moving picture decoding apparatus characterized by comprising:

  According to the present invention, the encoded data is decoded by the single-viewpoint decoding unit to generate the decoded images of the plurality of viewpoints, and the parameters used for the geometric transformation at the time of generating the encoded data It was decided to derive. Further, the inter-viewpoint processing means geometrically transforms the decoded image of the decoding-side reference viewpoint generated by the single-viewpoint decoding means using the parameters derived by the single-viewpoint decoding means, and after the decoding-side reference viewpoint is converted. The decoded image is generated. Further, the reference image list adding means serves as a reference image for decoding the decoded side reference viewpoint generated by the inter-viewpoint processing means and decoding the decoded side extended viewpoint encoded data by the single viewpoint decoding means. We decided to make it available.

  For this reason, a frame of another layer is added to the reference image list, and encoded data can be decoded using a framework similar to MV-HEVC disclosed in Non-Patent Document 1, so that video The change below the encoding layer is not necessary, and the existing HEVC codec design can be utilized to the maximum extent. Therefore, the mounting cost can be suppressed.

  Further, the decoded image of the decoding side reference viewpoint is geometrically transformed according to the relationship between the decoding side reference viewpoint and the decoding side extended viewpoint to generate a decoded image after conversion of the decoding side reference viewpoint. Using the decoded image after the viewpoint conversion, the encoded data of the decoding-side extended viewpoint can be decoded. Here, a multi-view video obtained by photographing the same subject has a feature that a moving image for each viewpoint can be converted into a similar moving image by geometric conversion. For this reason, since a similar image can be used as a reference image by geometric transformation according to the relationship between the decoding-side reference viewpoint and the decoding-side extended viewpoint, the encoding performance can be improved.

  (8) The present invention relates to the moving picture decoding apparatus according to (7), wherein the single-viewpoint decoding unit decodes the encoded data and is used for illuminance compensation when the encoded data is generated. Parameters (e.g., corresponding to illuminance compensation parameters a and b described later) are also derived, and the inter-viewpoint processing unit obtains the decoding obtained when the single-viewpoint decoding unit decodes the moving image of the decoding-side reference viewpoint. A moving image characterized in that a decoded image of a side reference viewpoint is subjected to geometric transformation and illuminance compensation using a parameter derived by the single-viewpoint decoding means to generate a decoded image after conversion of the decoding side reference viewpoint A decoding device is proposed.

  According to the present invention, in the moving picture decoding apparatus according to (7), the encoded data is decoded by the single-viewpoint decoding means, and the parameters used in the illumination compensation at the time of generating the encoded data are also derived. It was. Further, the inter-viewpoint processing means decodes the decoding image of the decoding-side reference viewpoint obtained when the moving image of the decoding-side reference viewpoint is decoded by the single-viewpoint decoding means using the parameters derived by the single-viewpoint decoding means. The decoded image after conversion of the decoding side reference viewpoint is generated by performing conversion and illumination compensation. Therefore, when the encoded data is decoded, the geometric conversion can be performed using the geometric conversion parameters and the illumination compensation parameters used in the moving image encoding apparatus.

  (9) In the moving image decoding apparatus according to (7) or (8), the inter-viewpoint processing unit performs either a projective transformation or an affine transformation by triangular patch division as a geometric transformation. Has been proposed.

  According to the present invention, in the video decoding device according to (7) or (8), either the projective transformation or the affine transformation based on the triangular patch division can be performed as the geometric transformation by the inter-viewpoint processing means. .

  (10) In the moving image decoding device according to any one of (7) to (9), the present invention provides the reference image list adding means after conversion of the decoding-side reference viewpoint generated by the inter-viewpoint processing means. A moving picture decoding apparatus is proposed in which a decoded picture can be used as a long-term reference frame when the moving picture at the decoding side extended viewpoint is decoded by the single-view decoding means.

  According to this invention, in the moving image decoding device according to any one of (7) to (9), the decoded image after conversion of the decoding-side reference viewpoint generated by the inter-viewpoint processing means by the reference image list adding means, The decoding side extended viewpoint video is made available as a long-term reference frame for decoding by the single viewpoint decoding means. This eliminates the need to recalculate the motion vector at the reference viewpoint, thereby reducing the amount of decoding processing.

  (11) The present invention relates to a moving image encoding device (e.g., corresponding to the moving image encoding device 1 in FIG. 1) that encodes moving images of a plurality of viewpoints with different perspectives to generate encoded data, and the moving image. A moving image processing system (for example, the moving image processing system in FIG. 1) including a moving image decoding device (for example, corresponding to the moving image decoding device 100 in FIG. 1) that decodes encoded data generated by the image encoding device. The video encoding device is a single-viewpoint encoding unit (for example, the first encoding unit in FIG. 2) that encodes each of the video images of the plurality of viewpoints for each viewpoint. 10 and the second encoding unit 20), one of the plurality of viewpoints as a reference viewpoint, and one of the plurality of viewpoints excluding the reference viewpoint as an extended viewpoint, The single viewpoint code is used as the reference viewpoint video. The local decoded image of the reference viewpoint (for example, equivalent to the local decoded image SIG3 of the reference viewpoint in FIG. 2) obtained by encoding by means is geometrically transformed according to the relationship between the reference viewpoint and the extended viewpoint. Then, encoding side inter-viewpoint processing means (for example, between the viewpoints in FIG. 2) that generates the local decoded image after the conversion of the reference viewpoint (for example, the local decoded image SIG4 after the conversion of the reference viewpoint in FIG. 2). And a local decoded image after conversion of the reference viewpoint generated by the encoding inter-viewpoint processing unit, and a moving image of the extended viewpoint by the single-viewpoint encoding unit. Encoding side reference image list addition means (for example, corresponding to the inter-viewpoint processing unit 30 in FIG. 2) that can be used as a reference image of the video, and the video decoding device decodes the encoded data , The decoded images of each of the plurality of viewpoints (for example, the decoded image SIG103 of the extended viewpoint in FIG. 5 and the decoded images SIG101 and SIG104 of the reference viewpoint) are generated, and the geometrical image is generated in the moving image encoding apparatus. Single-viewpoint decoding means (for example, the first decoding unit 110 and the second decoding unit 120 in FIG. 5) that derive parameters (e.g., corresponding to each element of the projective transformation matrix s described later) used in the conversion. And the decoded image of the reference viewpoint generated by the single-viewpoint decoding means (for example, the decoded image SIG101 of the reference viewpoint of FIG. 5) using the parameters derived by the single-viewpoint decoding means. Reconstruction that generates a decoded image after conversion of the reference viewpoint (for example, equivalent to the decoded image SIG102 after conversion of the reference viewpoint in FIG. 5) by performing geometric conversion. The inter-side-view processing means (e.g., corresponding to the inter-view processing section 130 in FIG. 5) and the decoded image after the conversion of the reference viewpoint generated by the decoding-side inter-view processing means is used as the extended viewpoint moving image. Moving image processing comprising: a decoding-side reference image list adding means (for example, corresponding to the inter-viewpoint processing unit 130 in FIG. 5) that can be used as a reference image for decoding by the single-viewpoint decoding means A system is proposed.

  According to the present invention, the same effects as described above can be obtained.

  (12) The present invention relates to single-viewpoint encoding means (for example, equivalent to the first encoding unit 10 and the second encoding unit 20 in FIG. 2) and inter-viewpoint processing means (for example, inter-viewpoint processing in FIG. 2). And a reference image list adding means (e.g., corresponding to the inter-viewpoint processing unit 30 in FIG. 2), and encodes moving images from a plurality of viewpoints with different perspectives to generate encoded data. A video decoding method in an encoding device (e.g., corresponding to the video encoding device 1 in FIG. 1), wherein the single-viewpoint encoding means uses one of the plurality of viewpoints as a reference viewpoint, and A first step of encoding a moving image of the reference viewpoint using one of a plurality of viewpoints excluding the reference viewpoint as an extended viewpoint, and the inter-viewpoint processing unit converts the moving image of the reference viewpoint Obtained when encoding by the first step. The local decoded image of the reference viewpoint (e.g., equivalent to the local decoded image SIG3 of the reference viewpoint in FIG. 2) is subjected to geometric transformation according to the relationship between the reference viewpoint and the extended viewpoint, and after the conversion of the reference viewpoint A second step of generating a local decoded image (e.g., corresponding to the local decoded image SIG4 after conversion of the reference viewpoint in FIG. 2), and the reference image list adding means is generated by the second step. A third step of making the local decoded image after conversion of the reference viewpoint available as a reference image; and the single-viewpoint encoding means uses the reference image made available by the third step, and And a fourth step of encoding a moving image of an extended viewpoint, and proposes a moving image encoding method.

  According to the present invention, the same effects as described above can be obtained.

  (13) The present invention provides a single-viewpoint decoding unit (for example, equivalent to the first decoding unit 110 and the second decoding unit 120 in FIG. 5) and an inter-viewpoint processing unit (for example, the inter-viewpoint processing unit 130 in FIG. 5). And a reference image list adding means (e.g., corresponding to the inter-viewpoint processing unit 130 in FIG. 5), and a moving image for decoding encoded data obtained by encoding moving images of a plurality of viewpoints with different perspectives. A video decoding method in an image decoding device (e.g., corresponding to the video decoding device 100 in FIG. 1), wherein the single-view decoding means uses a reference viewpoint when generating the encoded data among the plurality of viewpoints. If the viewpoint that has been used is the decoding-side reference viewpoint and the viewpoint that was the extended viewpoint at the time of generation of the encoded data is the decoding-side extended viewpoint, the encoded data is decoded and the decoded image ( For example, the basis of FIG. The viewpoint decoded images SIG101 and SIG104) and parameters used for the geometric transformation at the time of generating the encoded data (for example, corresponding to each element of the projective transformation matrix s described later) are derived. The inter-viewpoint processing means, the decoded image of the decoding-side reference viewpoint generated by the first step (for example, corresponding to the decoded image SIG101 of the reference viewpoint in FIG. 5), Geometric transformation is performed using the parameters derived in the first step to generate a decoded image after conversion of the decoding-side reference viewpoint (for example, the decoded image SIG102 after conversion of the reference viewpoint in FIG. 5). A second step, wherein the reference image list adding means converts the decoded reference image generated after the second step into a reference image; A third step of making available as a fourth step, wherein the single-view decoding means decodes the encoded data of the decoding-side extended viewpoint using the reference image made available by the third step A video decoding method characterized by comprising: a step.

  According to the present invention, the same effects as described above can be obtained.

  (14) The present invention relates to single-viewpoint encoding means (for example, equivalent to the first encoding unit 10 and the second encoding unit 20 in FIG. 2) and inter-viewpoint processing means (for example, inter-viewpoint processing in FIG. 2). And a reference image list adding means (e.g., corresponding to the inter-viewpoint processing unit 30 in FIG. 2), and encodes moving images from a plurality of viewpoints with different perspectives to generate encoded data. A program for causing a computer to execute a moving image decoding method in an encoding device (e.g., corresponding to the moving image encoding device 1 in FIG. 1), wherein the single-viewpoint encoding means includes a plurality of viewpoints. A first step of encoding a moving image of the reference viewpoint using one of the plurality of viewpoints as an extended viewpoint and one of the plurality of viewpoints excluding the reference viewpoint, and the inter-viewpoint processing means Is the video of the reference viewpoint The local decoded image of the reference viewpoint (for example, equivalent to the local decoded image SIG3 of the reference viewpoint in FIG. 2) obtained when encoding is performed in the first step, is related to the relationship between the reference viewpoint and the extended viewpoint. And a second step of generating a local decoded image after conversion of the reference viewpoint (for example, corresponding to the local decoded image SIG4 after conversion of the reference viewpoint in FIG. 2), and adding the reference image list A third step in which the means makes it possible to use, as a reference image, the local decoded image after the transformation of the base viewpoint generated in the second step; and the single-viewpoint encoding means has the third step A program for causing a computer to execute a fourth step of encoding the extended viewpoint moving image using a reference image made available by .

  According to the present invention, the same effects as described above can be obtained.

  (15) The present invention provides a single-viewpoint decoding unit (for example, equivalent to the first decoding unit 110 and the second decoding unit 120 in FIG. 5) and an inter-viewpoint processing unit (for example, the inter-viewpoint processing unit 130 in FIG. 5). And a reference image list adding means (e.g., corresponding to the inter-viewpoint processing unit 130 in FIG. 5), and a moving image for decoding encoded data obtained by encoding moving images of a plurality of viewpoints with different perspectives. A program for causing a computer to execute a moving image decoding method in an image decoding device (e.g., corresponding to the moving image decoding device 100 in FIG. 1), wherein the single-viewpoint decoding unit includes: If the viewpoint that was the reference viewpoint at the time of generating the encoded data is the decoding side reference viewpoint, and the viewpoint that was the extended viewpoint at the time of generating the encoded data is the decoding side extended viewpoint, the encoded data is decoded, Recovery A side reference viewpoint decoded image (for example, corresponding to the reference viewpoint decoded images SIG101 and SIG104 in FIG. 5) is generated, and parameters used for geometric transformation at the time of generation of the encoded data (for example, A first step of deriving a projection transformation matrix s (to be described later) and a decoded image of the decoding-side reference viewpoint generated by the first step by the inter-viewpoint processing means (for example, FIG. 5). The decoded image SIG101 of the reference viewpoint is geometrically transformed using the parameters derived in the first step, and the decoded image after the conversion of the decoding-side reference viewpoint (for example, the reference viewpoint of FIG. 5). And the reference image list adding means is generated by the second step. A third step of making the decoded image after conversion of the decoding-side reference viewpoint usable as a reference image, and the single-view decoding means using the reference image made available by the third step, A program for causing a computer to execute the fourth step of decoding the encoded data of the decoding side extended viewpoint is proposed.

  According to the present invention, the same effects as described above can be obtained.

  According to the present invention, it is possible to improve the encoding efficiency of videos from a plurality of viewpoints with different perspectives while suppressing the mounting cost.

1 is a block diagram of a moving image processing system according to an embodiment of the present invention. It is a block diagram of the moving image encoder which concerns on one Embodiment of this invention. It is a block diagram of the 1st encoding part with which the moving image encoder which concerns on one Embodiment of this invention is provided. It is a block diagram of the 2nd encoding part with which the moving image encoder which concerns on one Embodiment of this invention is provided. It is a block diagram of the moving image decoding apparatus which concerns on one Embodiment of this invention. It is a block diagram of the 1st decoding part with which the moving image decoding apparatus which concerns on one Embodiment of this invention is provided. It is a block diagram of the 2nd decoding part with which the moving image decoding apparatus which concerns on one Embodiment of this invention is provided. It is a figure which shows the combination of a reference viewpoint and an extended viewpoint. It is a figure which shows the combination of a reference viewpoint and an extended viewpoint. It is a figure for demonstrating the reference image in MV-HEVC.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements, and various variations including combinations with other existing constituent elements are possible. Accordingly, the description of the following embodiments does not limit the contents of the invention described in the claims.

[Configuration and Operation of Moving Image Processing System AA]
FIG. 1 is a block diagram of a moving image processing system AA according to an embodiment of the present invention. The moving image processing system AA encodes moving images of a plurality of viewpoints with different perspectives to generate encoded data (see the extended viewpoint bit stream SIG6 and the reference viewpoint bit stream SIG7 in FIG. 2). The apparatus 1 and the moving image decoding apparatus 100 which decodes the encoding data produced | generated by the moving image encoding apparatus 1 are provided. These moving image encoding apparatus 1 and moving image decoding apparatus 100 transmit and receive the above-described encoded data via a transmission path, for example.

  In the present embodiment, there are two viewpoints, that is, a reference viewpoint and an extended viewpoint, as a plurality of viewpoints with different perspectives, and the reference viewpoint is a base layer and the extended viewpoint is an enhancement layer.

[Configuration and Operation of Video Encoding Device 1]
FIG. 2 is a block diagram of the moving picture encoding apparatus 1. The moving image encoding apparatus 1 includes a first encoding unit 10, a second encoding unit 20, and an inter-viewpoint processing unit 30. The first encoding unit 10 encodes the extended-view original image SIG1 and outputs it as an extended-viewpoint bitstream SIG6. The second encoding unit 20 encodes the original image SIG2 of the reference viewpoint and outputs it as a bit stream SIG7 of the reference viewpoint. The inter-viewpoint processing unit 30 generates a local decoded image SIG4 after conversion of the reference viewpoint by performing projective transformation and illuminance compensation on the local decoded image SIG3 of the reference viewpoint according to the relationship between the reference viewpoint and the extended viewpoint. The local decoded image SIG4 after the conversion of the first encoding unit 10 can be used as a reference image when the first encoding unit 10 performs inter prediction. The operations of the first encoding unit 10, the second encoding unit 20, and the inter-viewpoint processing unit 30 will be described in detail below.

  FIG. 3 is a block diagram of the first encoding unit 10. The first encoding unit 10 includes an inter prediction unit 11, an intra prediction unit 12, a transform / quantization unit 13, an entropy encoding unit 14, an inverse quantization / inverse transform unit 15, an in-loop filter unit 16, and a buffer unit. 17.

  The inter prediction unit 11 receives an extended viewpoint original image SIG1 and a filtered local decoded image SIG18 (described later) supplied from the buffer unit 17. The inter prediction unit 11 performs inter prediction using the extended viewpoint original image SIG1 and the filtered local decoded image SIG18 to generate and output an inter predicted image SIG11.

  The intra prediction unit 12 receives an extended viewpoint original image SIG1 and an after-filter local decoded image SIG16 described later. The intra prediction unit 12 performs intra prediction using the extended viewpoint original image SIG1 and the pre-filter local decoded image SIG16 to generate and output an intra predicted image SIG12.

  The transform / quantization unit 13 receives an error (residual) signal SIG13 between the extended viewpoint original image SIG1 and the inter predicted image SIG11 or the intra predicted image SIG12. The transform / quantization unit 13 transforms and quantizes the input residual signal SIG13 to generate a quantized coefficient SIG14 and outputs it.

  The entropy encoding unit 14 receives a quantization coefficient SIG14, a transformation parameter SIG5 (described later), and side information (not shown) (related information such as a prediction mode and a motion vector necessary for pixel value reconstruction) as inputs. The entropy encoding unit 14 performs entropy encoding on the input signal and outputs it as an extended viewpoint bit stream SIG6.

  The inverse quantization / inverse transform unit 15 receives the quantization coefficient SIG14. The inverse quantization / inverse transform unit 15 inversely quantizes and inversely transforms the quantization coefficient SIG14 to generate and output an inversely transformed residual signal SIG15.

  The in-loop filter unit 16 receives the pre-filter local decoded image SIG16. The pre-filter local decoded image SIG16 is a signal obtained by adding up the inter prediction image SIG11 or the intra prediction image SIG12 and the inversely converted residual signal SIG15. The in-loop filter unit 16 applies an in-loop filter such as a deblocking filter to the pre-filter local decoded image SIG16 to generate and output a post-filter local decoded image SIG17.

  The buffer unit 17 accumulates the filtered local decoded image SIG17 and the local decoded image SIG4 after conversion of the reference viewpoint, and appropriately supplies the filtered local decoded image SIG18 to the inter prediction unit 11 as the filtered local decoded image SIG18.

  FIG. 4 is a block diagram of the second encoding unit 20. The second encoding unit 20 includes an inter prediction unit 21, an intra prediction unit 22, a transform / quantization unit 23, an entropy encoding unit 24, an inverse quantization / inverse transform unit 25, an in-loop filter unit 26, and a buffer unit. 27.

  The inter prediction unit 21, the intra prediction unit 22, the transform / quantization unit 23, the entropy encoding unit 24, the inverse quantization / inverse transform unit 25, and the in-loop filter unit 26 are respectively the inter prediction unit 11, FIG. It operates in the same manner as the intra prediction unit 12, transform / quantization unit 13, entropy coding unit 14, inverse quantization / inverse transform unit 15, and in-loop filter unit 16. On the other hand, the buffer unit 27 performs an operation different from that of the buffer unit 17 of FIG.

  The buffer unit 27 accumulates the filtered local decoded image SIG27 output from the in-loop filter unit 26, and appropriately supplies the filtered local decoded image SIG28 as the filtered local decoded image SIG28 to the inter prediction unit 21, and as the local decoded image SIG3 of the reference viewpoint Output.

  Returning to FIG. 2, the inter-viewpoint processing unit 30 receives the expanded viewpoint original image SIG1, the reference viewpoint original image SIG2, and the reference viewpoint local decoded image SIG3. The inter-viewpoint processing unit 30 performs projective transformation matrix derivation processing, illuminance compensation parameter derivation processing, and conversion processing.

  First, the projective transformation matrix derivation process will be described. In the projective transformation matrix derivation process, the inter-viewpoint processing unit 30 first uses the feature point detection algorithm such as a SIFT (Scale-Invariant Feature Transform) algorithm to perform a feature from the expanded viewpoint original image SIG1 and the reference viewpoint original image SIG2. A point is extracted, and a degree of coincidence between each feature point in the original image SIG1 at the extended viewpoint and each feature point in the original image SIG2 at the reference viewpoint is calculated. Next, four sets of feature points are identified as the corresponding feature points between the original image SIG1 of the extended viewpoint and the original image SIG2 of the reference viewpoint in order from the one with the highest degree of coincidence. Next, an 8-dimensional simultaneous equation with the elements of the projective transformation matrix as variables (8 unknowns) is set up to derive the projective transformation matrix s (for example, <Jan Erik Solem, Aikawa Aizo, "Practical Computer Vision" See O'Reilly Japan, Inc., March 2013>).

  Next, the illumination compensation parameter derivation process will be described. In the illumination compensation parameter derivation process, the inter-viewpoint processing unit 30 first assumes a linear prediction formula expressed by the following formula (1). Next, the luminance value of the expanded viewpoint original image SIG1 is substituted into the vector y of the equation (1), and the luminance value of the reference viewpoint original image SIG2 is substituted into the vector x of the equation (1). Scalars a and b satisfying 1) are derived by the least square method and set as illumination compensation parameters a and b.

  Note that the luminance value of the extended viewpoint original image SIG1 to be substituted into the vector y may be the luminance values of all the pixels constituting the extended viewpoint original image SIG1, or all of the elements constituting the extended viewpoint original image SIG1. May be a luminance value of a pixel in a predetermined region (for example, the central portion) of the pixels, or a predetermined pixel (for example, one pixel) of all the pixels constituting the expanded viewpoint original image SIG1. The luminance value of the pixel after thinning out every other pixel may be used. Further, the luminance value of the reference viewpoint original image SIG2 to be substituted into the vector x may be the luminance values of all the pixels constituting the reference viewpoint original image SIG2, or all of the reference viewpoint original image SIG2 constituting the reference viewpoint. May be a luminance value of a pixel in a predetermined region (for example, the central portion) of the pixels, or a predetermined pixel (for example, one pixel) among all the pixels constituting the reference viewpoint original image SIG2. The luminance value may be obtained by thinning out every other pixel. In addition, the luminance value of the local decoded image SIG3 of the reference viewpoint may be substituted for the vector x in Expression (1) instead of the luminance value of the original image SIG2 of the reference viewpoint.

  Each element of the projective transformation matrix s obtained by the projective transformation matrix derivation process and the illuminance compensation parameters a and b obtained by the illuminance compensation parameter derivation process are converted into the first encoding unit 10 as the above-described transformation parameter SIG5. To the entropy encoding unit 14.

  Here, the projective transformation matrix s is a 3 × 3 matrix, and one of the nine elements is always “1”. For this reason, each element of the above-described projective transformation matrix s is eight elements excluding “1” among the nine elements of the projective transformation matrix s, and these eight elements are first encoded as scalars. It is transmitted to the entropy encoding unit 14 of the unit 10.

  The projective transformation matrix s can also be expressed as the coordinates of four points in the image at the extended viewpoint corresponding to the corner of the image at the reference viewpoint. Here, the derivation of the projective transformation matrix s is performed by both the moving image coding apparatus 1 and the moving image decoding apparatus 100. The moving image decoding apparatus 100 performs a feature point detection algorithm such as a SIFT algorithm. I can't. For this reason, it is good also as transmitting the coordinate of the four said points | pieces to the entropy encoding part 14 of the 1st encoding part 10 instead of the above-mentioned eight elements.

  Next, the conversion process will be described. In the conversion process, the inter-viewpoint processing unit 30 applies the projection conversion matrix s to the local decoded image SIG3 of the reference viewpoint and performs the projection conversion, and then applies the illuminance compensation parameters a and b to perform the illuminance compensation. Then, the local decoded image SIG4 after the conversion of the base viewpoint is generated and stored as a long-term reference frame in the buffer unit 17 of the first encoding unit 10.

[Configuration and Operation of Video Decoding Device 100]
FIG. 5 is a block diagram of the video decoding device 100. The moving picture decoding apparatus 100 includes a first decoding unit 110, a second decoding unit 120, and an inter-viewpoint processing unit 130. The first decoding unit 110 decodes the extended-viewpoint bitstream SIG6 and outputs it as an extended-viewpoint decoded image SIG103. The second decoding unit 120 decodes the reference viewpoint bit stream SIG7 and outputs it as a reference viewpoint decoded image SIG104. The inter-viewpoint processing unit 130 performs the projective conversion and the illuminance compensation on the decoded image SIG101 of the reference viewpoint using the projection transformation matrix s and the illuminance compensation parameters a and b derived by the video encoding device 1, and the reference viewpoint The decoded image SIG102 after the conversion is generated, and the decoded image SIG102 after the base viewpoint conversion can be used as a reference image when the first decoding unit 110 performs the inter prediction. The operations of the first decoding unit 110, the second decoding unit 120, and the inter-viewpoint processing unit 130 will be described in detail below.

  FIG. 6 is a block diagram of the first decoding unit 110. The first decoding unit 110 includes an entropy decoding unit 111, an inverse transform / inverse quantization unit 112, an inter prediction unit 113, an intra prediction unit 114, an in-loop filter unit 115, and a buffer unit 116.

  The entropy decoding unit 111 receives the extended view bit stream SIG6. The entropy decoding unit 111 entropy-decodes the bit stream SIG6 of the extended viewpoint, the quantization coefficient level SIG111, the transformation parameter SIG5 generated by the video encoding device 1, and side information (necessary for pixel value reconstruction) Related information such as a prediction mode and a motion vector).

  The inverse transform / inverse quantization unit 112 receives the quantization coefficient level SIG111. The inverse transform / inverse quantization unit 112 performs inverse transform and inverse quantization on the quantized coefficient level SIG111 to generate and output an inversely transformed residual signal SIG112.

  The inter prediction unit 113 receives a filtered local decoded image SIG117 described later supplied from the buffer unit 116 as an input. The inter prediction unit 113 performs inter prediction using the filtered local decoded image SIG117 to generate and output an inter predicted image SIG113.

  The intra prediction unit 114 receives the pre-filter local decoded image SIG115 as input. The pre-filter local decoded image SIG115 is a signal obtained by summing the inversely transformed residual signal SIG112 and the inter predicted image SIG113 or the intra predicted image SIG114. The intra prediction unit 114 performs intra prediction using the pre-filter local decoded image SIG115 to generate and output an intra predicted image SIG114.

  The in-loop filter unit 115 receives the pre-filter local decoded image SIG115 as input. The in-loop filter unit 115 applies an in-loop filter such as a deblocking filter to the pre-filter local decoded image SIG 115 to generate and output a post-filter local decoded image SIG 116.

  The buffer unit 116 accumulates the decoded image SIG102 after conversion of the reference viewpoint and the filtered local decoded image SIG116, and supplies the filtered image to the inter prediction unit 113 as a filtered local decoded image SIG117 as appropriate. The decoded image SIG103 is output.

  FIG. 7 is a block diagram of the second decoding unit 120. The second decoding unit 120 includes an entropy decoding unit 121, an inverse transform / inverse quantization unit 122, an inter prediction unit 123, an intra prediction unit 124, an in-loop filter unit 125, and a buffer unit 126.

  The entropy decoding unit 121, the inverse transform / inverse quantization unit 122, the inter prediction unit 123, the intra prediction unit 124, and the in-loop filter unit 125 are respectively the entropy decoding unit 111, the inverse transform / inverse quantization unit 112 in FIG. , The inter prediction unit 113, the intra prediction unit 114, and the in-loop filter unit 115 operate in the same manner. On the other hand, the buffer unit 126 performs an operation different from that of the buffer unit 116 of FIG.

  The buffer unit 126 accumulates the filtered local decoded image SIG126 output from the in-loop filter unit 125, and appropriately supplies the filtered local decoded image SIG127 to the inter prediction unit 123 as the filtered local decoded image SIG127. Output as SIG104.

  Returning to FIG. 5, the inter-viewpoint processing unit 130 receives the transformation parameter SIG5 generated by the video encoding device 1 and the decoded image SIG101 of the reference viewpoint. The inter-viewpoint processing unit 130 applies projection conversion matrix s included in the conversion parameter SIG5 to the decoded image SIG101 of the reference viewpoint, and then performs illuminance compensation parameter a included in the conversion parameter SIG5. , B is applied to perform illuminance compensation to generate a decoded image SIG102 after conversion of the base viewpoint, and store it as a long-term reference frame in the buffer unit 116 of the first decoding unit 110.

  According to the above moving picture coding apparatus 1, the following effects can be produced.

  The moving image encoding apparatus 1 adds a frame of the base viewpoint to the reference image list of the extended viewpoint. For this reason, since a moving image can be encoded using the same framework as MV-HEVC shown in Non-Patent Document 1, it is not necessary to change the video encoding layer and below, and the existing HEVC The codec design can be diverted to the maximum extent. Therefore, the mounting cost can be suppressed.

  In addition, the moving image encoding apparatus 1 generates a local decoded image SIG4 after conversion of the reference viewpoint by performing projective conversion and illumination compensation on the local decoded image SIG3 of the reference viewpoint according to the relationship between the reference viewpoint and the extended viewpoint. Then, the original image SIG1 of the extended viewpoint can be encoded using the local decoded image SIG4 after the conversion of the reference viewpoint. Here, a multi-view video obtained by photographing the same subject has a feature that a moving image for each viewpoint can be converted into a similar moving image by projective transformation. For this reason, a similar image can be used as a reference image by projective transformation according to the relationship between the base viewpoint and the extended viewpoint, so that the encoding performance can be improved.

  Further, as described above, the moving image encoding device 1 performs the projective conversion and the illuminance compensation on the local decoded image SIG3 of the reference viewpoint according to the relationship between the reference viewpoint and the extended viewpoint, and converts the local viewpoint after the conversion of the reference viewpoint. The decoded image SIG4 is generated, and the original image SIG1 of the extended viewpoint can be encoded using the local decoded image SIG4 after the conversion of the reference viewpoint. For this reason, even if the illuminance differs between the reference viewpoint original image SIG2 and the extended viewpoint original image SIG1, even if the illuminance is different between the reference viewpoint and the extended viewpoint, the shooting environment such as the illumination condition is different. Can compensate for the difference. Therefore, even when the illuminance differs between the reference viewpoint original image SIG2 and the expanded viewpoint original image SIG1, the encoding performance can be improved by illuminance compensation according to the relationship between the reference viewpoint and the expanded viewpoint.

  In addition, the moving image encoding device 1 transmits each element of the projective transformation matrix s and the illumination compensation parameters a and b to the moving image decoding device 100. Therefore, the moving picture decoding apparatus 100 can perform the same projection conversion and illuminance compensation as the moving picture encoding apparatus 1 by using the projection transformation matrix s and the illuminance compensation parameters a and b.

  Also, the moving image encoding apparatus 1 causes the inter-viewpoint processing unit 30 to store the local decoded image SIG4 after the base viewpoint conversion in the buffer unit 17 as a long-term reference frame. This eliminates the need to recalculate the motion vector at the reference viewpoint, thereby reducing the amount of encoding processing.

  According to the above video decoding device 100, the following effects can be obtained.

  The moving image decoding apparatus 100 adds the frame of the standard viewpoint to the extended viewpoint reference image list. For this reason, since encoded data can be decoded using the same framework as MV-HEVC shown in Non-Patent Document 1, it is not necessary to change the video encoding layer and below, and existing HEVC The codec design can be diverted to the maximum extent. Therefore, the mounting cost can be suppressed.

  Further, the moving picture decoding apparatus 100 generates a decoded image SIG102 after the conversion of the reference viewpoint by performing projection conversion and illumination compensation on the decoded image SIG101 of the reference viewpoint according to the relationship between the reference viewpoint and the extended viewpoint. The extended view bitstream SIG6 can be decoded using the decoded image SIG102 after the conversion of the reference view. Here, a multi-view video obtained by photographing the same subject has a feature that a moving image for each viewpoint can be converted into a similar moving image by projective transformation. For this reason, a similar image can be used as a reference image by projective transformation according to the relationship between the base viewpoint and the extended viewpoint, so that the encoding performance can be improved.

  In addition, the moving image decoding apparatus 100 causes the buffer unit 116 to accumulate the decoded image SIG102 after the base viewpoint conversion as a long-term reference frame. This eliminates the need to recalculate the motion vector at the reference viewpoint, thereby reducing the amount of decoding processing.

  Note that the processing of the moving image encoding device 1 and the moving image decoding device 100 of the present invention is recorded on a computer-readable non-transitory recording medium, and the program recorded on the recording medium is recorded as the moving image encoding device 1. Alternatively, the present invention can be realized by being read and executed by the video decoding device 100.

  Here, for example, a nonvolatile memory such as an EPROM or a flash memory, a magnetic disk such as a hard disk, a CD-ROM, or the like can be applied to the above-described recording medium. In addition, reading and execution of the program recorded on the recording medium is performed by a processor provided in the moving image encoding device 1 or the moving image decoding device 100.

  Further, the above-described program is transferred from the moving image encoding device 1 or the moving image decoding device 100 storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. May be transmitted. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  Further, the above-described program may be for realizing a part of the above-described function. Furthermore, what can implement | achieve the above-mentioned function in combination with the program already recorded on the moving image encoder 1 or the moving image decoder 100, what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design that does not depart from the gist of the present invention.

  For example, in the above-described embodiment, the inter-viewpoint processing unit 30 performs the projective transformation by applying the projective transformation matrix s to the local decoded image SIG3 of the reference viewpoint in the transformation process, and then the illumination compensation parameter a, Illuminance compensation was performed by applying b. However, the present invention is not limited thereto, and the projection transformation may be performed by applying the projection transformation matrix s to the local decoded image SIG3 of the reference viewpoint after performing the illumination compensation by applying the illumination compensation parameters a and b. Good. Further, similarly to the inter-view processing unit 30, the inter-view processing unit 130 performs illuminance compensation by applying the illuminance compensation parameters a and b to the decoded image SIG 101 of the reference viewpoint, and then performs a projective transformation matrix. Projection transformation may be performed by applying s.

  In the above-described embodiment, the projective transformation is performed as the geometric transformation. However, the present invention is not limited to this. For example, affine transformation by triangular patch division may be performed as the geometric transformation.

  In the above-described embodiment, the inter-view processing unit 30 accumulates the local decoded image SIG4 after conversion of the base viewpoint as a long-term reference frame in the buffer unit 17, and the inter-view processing unit 130 converts the base viewpoint after conversion of the base viewpoint. The decoded image SIG102 is stored in the buffer unit 116 as a long-term reference frame. In the above-described embodiment, the inter-view processing unit 30 stores the decoded image SIG102 after the base viewpoint conversion in the buffer unit 116 as a long-term reference frame. However, it may be stored as a short-term reference frame, not as a long-term reference frame.

  In the above-described embodiment, it is assumed that there are two viewpoints, the reference viewpoint and the extended viewpoint. However, the present invention is not limited to this, and three or more viewpoints may exist. For example, when there are three viewpoints, as shown in FIG. 8, the reference viewpoint when the viewpoint B is the extended viewpoint is the viewpoint A, and the reference viewpoint when the viewpoint C is the extended viewpoint is also the viewpoint A. Alternatively, as shown in FIG. 9, the reference viewpoint when the viewpoint B is an extended viewpoint may be the viewpoint A, and the reference viewpoint when the viewpoint C is the extended viewpoint may be the viewpoint B.

AA ... Moving image processing system 1 ... Moving image encoding device 10 ... First encoding unit 20 ... Second encoding unit 30 ... Inter-viewpoint processing unit 100 ... Movie Image decoding apparatus 110... First decoding unit 120... Second decoding unit 130.

Claims (15)

A moving image encoding device that generates encoded data by encoding moving images of a plurality of viewpoints with different perspectives,
Single-viewpoint encoding means for encoding each of the moving images of the plurality of viewpoints for each viewpoint;
One of the plurality of viewpoints is set as a reference viewpoint, and one of the plurality of viewpoints excluding the reference viewpoint is set as an extended viewpoint, and a moving image of the reference viewpoint is encoded by the single viewpoint encoding unit. Inter-view processing for generating a local decoded image after conversion of the reference viewpoint by geometrically transforming the local decoded image of the reference viewpoint obtained at the time of conversion into a reference according to the relationship between the reference viewpoint and the extended viewpoint Means,
Reference image list that makes it possible to use the local decoded image after conversion of the base viewpoint generated by the inter-viewpoint processing unit as a reference image when the moving image of the extended viewpoint is encoded by the single-viewpoint encoding unit A moving image encoding apparatus comprising: an adding unit;   2. The moving picture coding apparatus according to claim 1, wherein the inter-viewpoint processing means transmits the parameters used in the geometric transformation to a moving picture decoding apparatus that decodes the coded data.   The inter-viewpoint processing unit determines a local decoded image of the reference viewpoint obtained when the moving image of the reference viewpoint is encoded by the single-viewpoint encoding unit according to the relationship between the reference viewpoint and the extended viewpoint. 2. The moving image encoding apparatus according to claim 1, wherein a local decoded image after conversion of the reference viewpoint is generated by performing geometric conversion and illuminance compensation.   4. The inter-viewpoint processing means transmits a parameter used for geometric transformation and a parameter used for illuminance compensation to a moving picture decoding apparatus that decodes the encoded data. The moving image encoding apparatus described in 1.   5. The moving picture encoding apparatus according to claim 1, wherein the inter-viewpoint processing unit performs any one of projective transformation and affine transformation by triangular patch division as geometric transformation.   The reference image list adding unit refers to a long-term reference when the local viewpoint image generated by the inter-viewpoint processing unit after the conversion of the base viewpoint is encoded by the single-viewpoint encoding unit. 6. The moving picture coding apparatus according to claim 1, wherein the moving picture coding apparatus is usable as a frame. A video decoding device that decodes encoded data obtained by encoding video from a plurality of viewpoints with different perspectives,
A single-viewpoint decoding means for decoding the encoded data to generate a decoded image for each of the plurality of viewpoints, and for deriving parameters used in geometric transformation at the time of generation of the encoded data;
Of the plurality of viewpoints, when the viewpoint that was the reference viewpoint at the time of generating the encoded data is a decoding-side reference viewpoint, and the viewpoint that was the extended viewpoint at the time of generating the encoded data is a decoding-side extended viewpoint, the single viewpoint is Viewpoint for generating a decoded image after conversion of the decoding-side reference viewpoint by geometrically transforming the decoded image of the decoding-side reference viewpoint generated by the viewpoint decoding means using the parameters derived by the single-viewpoint decoding means Interprocessing means;
Reference that enables the decoded image after conversion of the decoding-side reference viewpoint generated by the inter-viewpoint processing unit to be used as a reference image when the encoded data of the decoding-side extended viewpoint is decoded by the single-viewpoint decoding unit A moving picture decoding apparatus comprising: an image list adding means; The single-viewpoint decoding means decodes the encoded data and derives a parameter used for illuminance compensation at the time of generation of the encoded data,
The inter-view processing means uses a parameter derived by the single-view decoding means for the decoded image of the decoding-side reference viewpoint obtained when the moving picture of the decoding-side reference viewpoint is decoded by the single-view decoding means. The moving picture decoding apparatus according to claim 7, wherein the decoded picture after conversion of the decoding-side reference viewpoint is generated by performing geometric conversion and illuminance compensation.   The moving image decoding apparatus according to claim 7 or 8, wherein the inter-viewpoint processing means performs any one of projective transformation and affine transformation by triangular patch division as geometric transformation.   The reference image list adding unit is a long-term decoding unit that decodes the decoded image of the decoding-side reference viewpoint generated by the inter-viewpoint processing unit and the moving image of the decoding-side extended viewpoint by the single-viewpoint decoding unit. 10. The moving picture decoding apparatus according to claim 7, wherein the moving picture decoding apparatus is usable as a reference frame. A moving image encoding device that encodes moving images from a plurality of viewpoints with different perspectives to generate encoded data, and a moving image decoding device that decodes encoded data generated by the moving image encoding device. A moving image processing system,
The moving image encoding device is:
Single-viewpoint encoding means for encoding each of the moving images of the plurality of viewpoints for each viewpoint;
One of the plurality of viewpoints is set as a reference viewpoint, and one of the plurality of viewpoints excluding the reference viewpoint is set as an extended viewpoint, and a moving image of the reference viewpoint is encoded by the single viewpoint encoding unit. Coding side that generates a local decoded image after conversion of the reference viewpoint by geometrically transforming the local decoded image of the reference viewpoint obtained at the time of conversion into a reference according to the relationship between the reference viewpoint and the extended viewpoint Inter-viewpoint processing means;
The local decoded image after conversion of the reference viewpoint generated by the encoding inter-viewpoint processing unit can be used as a reference image when the extended viewpoint moving image is encoded by the single-viewpoint encoding unit. Encoding side reference image list addition means,
The moving picture decoding device comprises:
A single-viewpoint decoding means for decoding the encoded data to generate a decoded image for each of the plurality of viewpoints, and for deriving parameters used in geometric transformation in the video encoding device;
A decoding-side viewpoint that generates a decoded image after conversion of the reference viewpoint by geometrically converting the decoded image of the reference viewpoint generated by the single-viewpoint decoding unit using the parameters derived by the single-viewpoint decoding unit Interprocessing means;
A decoding-side reference image that can be used as a reference image when the extended-view moving image is decoded by the single-view decoding unit, using the decoded image after the conversion of the base viewpoint generated by the decoding-side viewpoint processing unit. And a list adding means. A moving picture decoding method in a moving picture coding apparatus, comprising a single-viewpoint coding means, an inter-viewpoint processing means, and a reference picture list adding means, and coding coded moving pictures from a plurality of viewpoints with different perspectives to generate coded data There,
The single-viewpoint encoding means uses one of the plurality of viewpoints as a reference viewpoint, and uses one of the plurality of viewpoints excluding the reference viewpoint as an extended viewpoint. A first step of encoding;
The inter-viewpoint processing means determines the local decoded image of the reference viewpoint obtained when the moving image of the reference viewpoint is encoded in the first step according to the relationship between the reference viewpoint and the extended viewpoint. A second step of performing geometric transformation to generate a local decoded image after transformation of the reference viewpoint;
A third step in which the reference image list adding means makes the local decoded image after the conversion of the standard viewpoint generated in the second step usable as a reference image;
The moving image characterized in that the single-viewpoint encoding unit includes a fourth step of encoding the extended-viewpoint moving image using the reference image made available in the third step. Encoding method. Moving picture decoding in a moving picture decoding apparatus that decodes encoded data obtained by encoding moving pictures of a plurality of viewpoints with different perspectives, comprising a single-view decoding means, an inter-view processing means, and a reference image list adding means A method,
The single-viewpoint decoding means sets the viewpoint that was the reference viewpoint when the encoded data was generated among the plurality of viewpoints as a decoding-side reference viewpoint, and the viewpoint that was the extended viewpoint when the encoded data was generated as the decoding side As an extended viewpoint, the encoded data is decoded to generate a decoded image of the decoding-side reference viewpoint, and a parameter used for geometric transformation at the time of generation of the encoded data is derived. And the steps
The inter-view processing means geometrically transforms the decoded image of the decoding-side reference viewpoint generated in the first step using the parameters derived in the first step, and A second step of generating a transformed decoded image;
A third step in which the reference image list adding means makes the decoded image after conversion of the decoding-side standard viewpoint generated in the second step available as a reference image;
A moving image characterized in that the single-view decoding means includes a fourth step of decoding the encoded data of the decoding-side extended viewpoint using the reference image made available in the third step. Image decoding method. A moving picture decoding method in a moving picture coding apparatus that includes a single-viewpoint encoding unit, an inter-viewpoint processing unit, and a reference image list addition unit, and generates encoded data by encoding moving images of a plurality of viewpoints with different perspectives. A program for causing a computer to execute,
The single-viewpoint encoding means uses one of the plurality of viewpoints as a reference viewpoint, and uses one of the plurality of viewpoints excluding the reference viewpoint as an extended viewpoint. A first step of encoding;
The inter-viewpoint processing means determines the local decoded image of the reference viewpoint obtained when the moving image of the reference viewpoint is encoded in the first step according to the relationship between the reference viewpoint and the extended viewpoint. A second step of performing geometric transformation to generate a local decoded image after transformation of the reference viewpoint;
A third step in which the reference image list adding means makes the local decoded image after the conversion of the standard viewpoint generated in the second step usable as a reference image;
A program for causing a computer to execute a fourth step in which the single-viewpoint encoding unit encodes a moving image of the extended viewpoint using a reference image that has been made available in the third step. Moving picture decoding in a moving picture decoding apparatus that decodes encoded data obtained by encoding moving pictures of a plurality of viewpoints with different perspectives, comprising a single-view decoding means, an inter-view processing means, and a reference image list adding means A program for causing a computer to execute the method,
The single-viewpoint decoding means sets the viewpoint that was the reference viewpoint when the encoded data was generated among the plurality of viewpoints as a decoding-side reference viewpoint, and the viewpoint that was the extended viewpoint when the encoded data was generated as the decoding side As an extended viewpoint, the encoded data is decoded to generate a decoded image of the decoding-side reference viewpoint, and a parameter used for geometric transformation at the time of generation of the encoded data is derived. And the steps
The inter-view processing means geometrically transforms the decoded image of the decoding-side reference viewpoint generated in the first step using the parameters derived in the first step, and A second step of generating a transformed decoded image;
A third step in which the reference image list adding means makes the decoded image after conversion of the decoding-side standard viewpoint generated in the second step available as a reference image;
A program for causing the computer to execute a fourth step in which the single-viewpoint decoding unit decodes the encoded data of the decoding-side extended viewpoint using the reference image made available in the third step. .

Images