JP2009004941A - Multi-viewpoint image receiving method, multi-viewpoint image receiving device, and multi-viewpoint image receiving program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that when an encoded multi-viewpoint image signal is received and decoded, a buffer is required for sorting the signals by viewpoint and delay is therefore caused. <P>SOLUTION: Encoded data obtained by encoding a multi-viewpoint image signal are received by a separation unit 21 through a network etc., and separated into first encoded data in which subsidiary information (SEI) and the number V of viewpoints of the multi-viewpoint image signal are encoded respectively and second encoded data wherein a decoded image output order number (o) and the multi-viewpoint image signal are encoded respectively. Those encoded data are individually decoded. A decoded image management unit 211 specifies viewpoints with numbers (v) for respective decoded image signals stored in a decoded image buffer 210, and manages the output order of decoded images at the respective viewpoints with a number (d) to output images having the same value of numbers (d) of decoded image signals at the respective viewpoints while synchronizing the viewpoints with one another so that the images are output simultaneously or successively. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a multi-view image receiving method, a multi-view image receiving apparatus, and a multi-view image receiving program, and in particular, multi-view image encoded data obtained by encoding multi-view image signals taken from different viewpoints via a network or the like. The present invention relates to a multi-view image receiving method, a multi-view image receiving apparatus, and a multi-view image receiving program.

<Video coding system>
Currently, moving images on the time axis are handled as digital signal information. At that time, motion compensated prediction is used using redundancy in the time direction for the purpose of broadcasting, transmitting or storing information with high efficiency. Devices and systems that are compliant with a coding scheme such as MPEG (Moving Picture Experts Group) that performs coding compression using orthogonal transform such as discrete cosine transform using redundancy in the spatial direction have become widespread.

  The MPEG-2 video (ISO / IEC 13818-2) encoding system established in 1995 is defined as a general-purpose moving image compression encoding system, and supports interlaced scanned images in addition to progressive scanned images. Supports not only SDTV (standard definition images) but also HDTV (high definition images), and storage of DVDs (Digital Versatile Disks), which are optical discs, and magnetic tapes using D-VHS (registered trademark) digital VTRs. It is widely used as an application for media and digital broadcasting.

  In addition, MPEG-4 visual (ISO / IEC 14496-2) encoding method was standardized, aiming at higher encoding efficiency in applications such as network transmission and portable terminals, and was established as an international standard in 1998. It was done.

  Furthermore, in 2003, joint work of the International Technical Organization (ISO) and the International Electrotechnical Commission (IEC) Joint Technical Committee (ISO / IEC) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) -4 AVC / H.264 encoding method (ISO / IEC 1449-10, ITU-T H.264 standard number. This is hereinafter referred to as AVC / H.264 encoding method. Called the international standard. This AVC / H.264 encoding method achieves higher encoding efficiency than conventional encoding methods such as MPEG-2 video and MPEG-4 visual.

  In a P picture (forward prediction encoded image) of an encoding system such as MPEG-2 video or MPEG-4 visual, motion compensation prediction is performed only from the immediately preceding I picture (in-screen encoded image) or P picture in the display order. I was going. On the other hand, in the AVC / H.264 coding system, P pictures and B pictures (bi-predictive coded images) can use a plurality of pictures as reference pictures, and the most suitable one for each block is selected from these pictures. Select to perform motion compensation. Further, in addition to the preceding picture in the display order, a subsequent picture can be referred to in the already encoded display order. In addition, a B picture of an encoding method such as MPEG-2 video or MPEG-4 visual refers to one reference picture in the display order, one reference picture in the rear, or two reference pictures at the same time. The average value of the two pictures is a predicted picture, and the difference data between the target picture and the predicted picture is encoded.

  On the other hand, in the AVC / H.264 coding system, a B picture can be referred to for prediction regardless of the forward or backward, regardless of the forward or backward, without being restricted by the restriction of one forward and one backward in the display order. It was. Furthermore, it is possible to refer to the B picture as a reference picture.

  Further, in the AVC / H.264 encoding method, when the decoded image is output, the reference picture is rearranged from the encoding order to the decoded image output order (desired order when displayed on the output destination display device or the like). Both of the non-reference pictures must be stored in the memory, but a mechanism for managing the reference pictures and the non-reference pictures in a single memory called a decoded picture buffer has been introduced. The coded sequence (bit stream) is encoded with information indicating an output order called a picture order count for each image, and is numbered in the output order of the decoded images. It has been. Of the images decoded and stored in the decoded image buffer, the images with the smallest picture order count value are sequentially output. In addition, the picture order count is reset in an IDR picture (a picture that means that subsequent pictures can be normally decoded without using information of pictures preceding the picture in the coding order).

  Furthermore, the encoding mode for each picture (VOP) has been determined using a picture in MPEG-2 video and a video object plane (VOP) in MPEG-4 as one unit, but AVC / H.264 encoding is used. In the system, a slice is used as an encoding unit, and it is also possible to have a configuration in which different slices such as an I slice, a P slice, and a B slice are mixed in one picture.

  Further, in the AVC / H.264 encoding method, a VCL (Video Coding Layer) that performs encoding / decoding processing of video pixel signals (encoding mode, motion vector, DCT coefficient, etc.), NAL ( Network Abstraction Layer) is defined.

  An encoded bit string encoded by the AVC / H.264 encoding method is configured in units of NAL units that are a delimiter of NAL. The NAL unit includes a VCL NAL unit including data (encoding mode, motion vector, DCT coefficient, etc.) encoded by VCL, and a non-VCL NAL unit not including data generated by VCL. The non-VCL NAL unit includes an SPS (sequence parameter set) that includes parameter information related to coding of the entire sequence, and a PPS (picture parameter parameter) that includes parameter information related to picture coding. Set), SEI (supplementary additional information) and the like which are not essential for decoding data encoded by VCL.

  A header (head part) of each NAL unit always has a flag (forbidden_zero_bit) having a value of “0”, an identifier (nal_ref_idc) for identifying whether an SPS or PPS, or a slice serving as a reference picture is included, An identifier (nal_unit_type) for identifying the type of NAL unit is included. The nal_unit_type is defined to have any value from “1” to “5” in the case of the VCL NAL unit. For the non-VCL NAL unit, for example, the SEI is “6” and the SPS is “7”. , PPS is defined to have a value of “8”. On the decoding side, the type of the NAL unit can be identified by nal_unit_type which is an identifier for identifying the type of the NAL unit included in the header part of the NAL unit.

  The basic unit of encoding in the AVC / H.264 encoding method is a slice obtained by dividing a picture, and the VCL NAL unit is a slice unit. Therefore, a unit called an access unit in which several NAL units are combined is defined, and one encoded picture is included in one access unit.

<Multi-view image coding method>
On the other hand, in a twin-lens stereoscopic television, a left-eye image and a right-eye image captured from two different directions by two cameras are generated and displayed on the same screen to show a stereoscopic image. I have to. In this case, the left eye image and the right eye image are separately transmitted or recorded as independent images. However, this requires about twice as much information as a single two-dimensional image.

  Therefore, conventionally, a method has been proposed in which one of the left and right images is set as a main image and the other image (sub-image) information is information-compressed by a general compression encoding method to reduce the amount of information (for example, Patent Document 1). In the stereoscopic television image transmission method described in Patent Document 1, a relative position with high correlation in the other image is obtained for each small region, and the position shift amount (parallax vector) and a difference signal (prediction residual signal) are obtained. ). The difference signal is also transmitted and recorded because the image close to the sub-image can be restored using the main image and the amount of disparity and position shift, which is parallax information, but there is no main image such as the shadow of the object. This is because the sub-image information cannot be restored.

  In 1996, a stereo image encoding method called a multi-view profile was added to the MPEG-2 video (ISO / IEC 14496-2) encoding method, which is an international standard for single-view image encoding (ISO / IEC). IEC 14496-2 / AMD3). The MPEG-2 video multi-view profile is a two-layer encoding method that encodes the image for the left eye with the base layer and the image for the right eye with the enhancement layer, and motion compensated prediction using redundancy in the time direction In addition to discrete cosine transformation using redundancy in the spatial direction, encoding compression is performed using disparity compensation prediction using redundancy between viewpoints.

  Furthermore, a method using one system encoding path and decoding path when encoding / decoding a multi-viewpoint image has been proposed (for example, see Patent Document 2). In this method, on the encoding side (transmission side), image signals of a plurality of channels obtained by photographing from a plurality of directions as shown in FIG. Delay one field at a time. The control unit 304 outputs a switch signal having a frame period or a field period, and controls the multiplexer 302 so as to sequentially extract the signals of a plurality of channels from the sequential unit 301.

  The multiplexer 302 synthesizes the signals of a plurality of channels in time series by sequentially inserting them in units of channels for each frame or field, thereby forming one system of image signals. The encoding unit 303 encodes one system of image signals output from the multiplexer 302 and outputs an encoded bit string to the transmission path.

  On the decoding side (receiving side), the coded bit string is decoded by the decoding means 305 as shown in FIG. The control unit 308 controls the demultiplexer 306 by outputting a switch signal having a frame period or a field period from the decoded image signal. As a result, each frame or field of the decoded image signal is sequentially extracted from the demultiplexer 306 in units of channels and supplied to the synchronizer 307, where the frames or fields are sequentially delayed by one frame or one field and synchronized. It is output as a multi-channel image signal similar to that at the time of shooting.

JP-A 61-144191 Japanese Patent Laid-Open No. 10-243419

  However, in the conventional stereoscopic image encoding / decoding method and apparatus, it is necessary to define which channel is the left and right images of the left and right image signals in which the L and R channel images are alternately arranged in advance. Further, since it is assumed that the images of the L and R channels are alternately arranged, it is not possible to deal with cases where the frame rates of the respective viewpoints are different or when images are missing.

  Further, in the conventional stereoscopic image encoding / decoding method and apparatus, the image signals of the plurality of channels constituting the multi-view image signal at the time of encoding are sequentially arranged for each frame or one field. Since the system image signal is input to the encoding unit 303 and further encoded and rearranged in the encoding order by the encoding unit 303, a buffer for sequentially rearranging every frame or one field is necessary. In that case, a delay may occur.

  Also, at the time of decoding, the image signals decoded by the decoding means 305 are sequentially arranged and output for each frame or field, and then sequentially delayed by one frame or field, so that the viewpoint is synchronized. A buffer for rearrangement is required every time, and a delay may occur at that time. Therefore, even in a receiving apparatus that receives the encoded data encoded by the above-described conventional stereoscopic image encoding method and apparatus and decodes the encoded data using the conventional stereoscopic image decoding method and apparatus, it is An image buffer is required and the delay time cannot be shortened.

  The present invention has been made in view of the above points, and provides a multi-view image receiving method, a multi-view image receiving apparatus, and a multi-view image receiving program that reduce the buffer size and shorten the delay time in the receiving-side apparatus. The purpose is to do.

In order to achieve the above object, the first invention is a multi-view image signal including image signals of respective viewpoints respectively obtained from a plurality of set viewpoints, and the image signal of one viewpoint is actually transmitted from one viewpoint. Receiving encoded data obtained by encoding a multi-view image signal, which is an image signal obtained by photographing an image signal or an image signal generated as a virtual image taken from one viewpoint, and decoding the encoded data A viewpoint image receiving method,
Supplementary additional information indicating that the multi-view image signal is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint, and the output order of the decoded image at each viewpoint A first step of receiving encoded data in which a decoded image output order number o (where o is an integer equal to or greater than 0) and a multi-view image signal are encoded, each indicating a number d indicating The first encoded data in which the supplementary additional information and the number of viewpoints V of the multi-view image signal are encoded, the decoded image output order number o, and the multi-view image Supplementary additional information including the second step of separating the second encoded data in which the signal is encoded and the number of viewpoints V of the multi-view image signal by decoding the first encoded data A third step of generating Obtained by dividing the decoded decoded image output order number o by the number V of decoded viewpoints by integer arithmetic, and generating a decoded image output order number o. The quotient (an integer greater than or equal to 0) is calculated as a number d indicating the output order of the decoded image at each viewpoint, and the remainder of the division (an integer greater than or equal to 0 and less than V) is the number v that identifies each viewpoint. A fifth step of calculating, a sixth step of decoding the second encoded data to generate a decoded multi-view image signal, a seventh step of storing the decoded multi-view image signal in an image buffer, The decoded multi-viewpoint image signal is extracted from the image buffer according to the number d indicating the output order of the decoded image at each viewpoint and the number v specifying each viewpoint calculated in the fifth step, and decoding is performed. Multi-view image Synchronize with each other a decoded image signal at each viewpoint which constitutes the characterized in that it comprises an eighth step of outputting.

In order to achieve the above object, the third aspect of the invention is a multi-viewpoint image signal including an image signal of each viewpoint obtained from each of a plurality of set viewpoints. Receives encoded data obtained by encoding a multi-view image signal that is an image signal obtained by actually capturing an image from a viewpoint or an image signal generated as a virtual image from one viewpoint, and A multi-viewpoint image receiving device for decoding,
The supplementary additional information indicating that the multi-view image is encoded, the number V of the viewpoints of the multi-view image signal, the number v for identifying each viewpoint, and the output order of the decoded image at each viewpoint. A reception unit that receives encoded data in which a decoded image output order number o (o is an integer equal to or greater than 0) and a multi-view image signal are respectively encoded, and the reception unit receives First encoded data in which supplementary additional information indicating that a multi-view image is encoded and the number of viewpoints V of the multi-view image signal are encoded from the encoded data, and decoded image output Separation means for separating the second encoded data in which the sequence number o and the multi-view image signal are encoded respectively, and the viewpoint of the multi-view image signal by decoding the separated first encoded data Generate supplementary information including the number V The first decoding means, the second decoding means for decoding the separated second encoded data to generate the decoded image output order number o, and the decoded decoded image output order number o is decoded. The quotient (integer) obtained by dividing the number of viewpoints V by integer arithmetic is calculated as a number d indicating the output order of the decoded image at each viewpoint, and the remainder of the division (an integer not less than 0 and less than V) Calculating means for calculating each of the viewpoints as a number v, third decoding means for decoding the second encoded data to generate a decoded multi-view image signal, and the decoded multi-view image signal as an image According to the storage means for storing in the buffer, the number d indicating the output order of the decoded image at each viewpoint supplied from the calculation means, and the number v for identifying each viewpoint, the decoded multi-view image signal from the image buffer Take out and decode multi-viewpoint Synchronize with each other a decoded image signal at each viewpoint to construct an image signal and having an output means for outputting. In order to achieve the above object, a multi-viewpoint image receiving program according to a fifth aspect of the present invention causes a computer to execute each step of the first aspect.

  In the first, third, and fifth inventions described above, supplementary additional information indicating that the multi-view image signal is encoded from the received encoded data, and the number of viewpoints V of the multi-view image signal, , Encoded image output order number o that collectively indicates a number v that identifies each viewpoint, and a number d that indicates the output order of the decoded image at each viewpoint. The multi-view image signal is separated from the second encoded data encoded, and the decoded multi-view image signal obtained by decoding each of the first and second encoded data is appropriately converted into an image buffer. The decoded multi-viewpoint image signal is extracted from the image buffer in accordance with the number d indicating the output order of the decoded image at each viewpoint obtained by decoding and calculation and the number v specifying each viewpoint. Decoded multi-view image No. By synchronously outputs together decoded image signal at each viewpoint which constitutes the, can be made unnecessary process and device for synchronization from the output of sequentially arranged for every one frame or one field after decoding.

  In the first, third, and fifth inventions, information indicating a multi-view image is encoded as SEI (supplementary additional information) together with the number of viewpoints V of the multi-view image signal, and the multi-view image Encoded data obtained by encoding a decoded image output order number o that collectively indicates a viewpoint number v that specifies the viewpoint of the image signal of each viewpoint that constitutes the signal and a number d that indicates the output order of the decoded image at each viewpoint Can be received and decoded.

In order to achieve the above object, the second invention is a multi-viewpoint image signal including image signals of respective viewpoints respectively obtained from a plurality of set viewpoints, and an image signal of one viewpoint is Receives encoded data obtained by encoding a multi-view image signal that is an image signal obtained by actually capturing an image from a viewpoint or an image signal generated as a virtual image from one viewpoint, and A multi-viewpoint image receiving method for decoding,
Supplementary additional information indicating that the multi-view image is encoded, the number V of the viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the viewpoint for managing the decoded image Decoded image output order number o (where o is 0 or more) that collectively indicates a correspondence relationship with the identified viewpoint ID, a number v that identifies each viewpoint, and a number d that indicates the output order of the decoded image at each viewpoint. Integer) and the encoded data received in the first step, the first step of receiving the encoded data in which the multi-view image signal is encoded, and the encoded data received in the first step. First encoded data in which the number V of viewpoints and the correspondence between a viewpoint v and a number v that specifies each viewpoint of the multi-viewpoint image are encoded, a decoded image output order number o, and a multi-viewpoint image Each signal is encoded A second step of separating the second encoded data, and decoding the first encoded data to specify the number V of viewpoints of the multi-view image signal and each viewpoint of the multi-view image A third step of generating supplementary additional information including the correspondence between the number v to be performed and the viewpoint ID, a fourth step of decoding the second encoded data to generate a decoded image output order number o, The quotient (integer) obtained by dividing the decoded decoded image output order number o by the number of decoded viewpoints V by integer arithmetic is calculated as a number d indicating the output order of the decoded images at the respective viewpoints. The fifth step of calculating the remainder of division (an integer greater than or equal to 0 and less than V) as a number v for identifying each viewpoint, and decoding the second encoded data to generate a decoded multi-view image signal Sixth step and decoding multi-viewpoint image signal In accordance with the number d indicating the output order of the decoded image at each viewpoint and the viewpoint ID calculated in the seventh step and the fifth step stored in the image buffer, the decoded multi-viewpoint image is output from the image buffer. And an eighth step of extracting the signal and outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other.

In order to achieve the above object, the fourth invention is a multi-viewpoint image signal including image signals of respective viewpoints respectively obtained from a plurality of set viewpoints. Receives encoded data obtained by encoding a multi-view image signal that is an image signal obtained by actually capturing an image from a viewpoint or an image signal generated as a virtual image from one viewpoint, and A multi-viewpoint image receiving device for decoding,
Supplementary additional information indicating that the multi-view image is encoded, the number V of the viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the viewpoint for managing the decoded image Decoded image output order number o (where o is 0 or more) that collectively indicates a correspondence relationship with the identified viewpoint ID, a number v that identifies each viewpoint, and a number d that indicates the output order of the decoded image at each viewpoint. An integer) and multi-view image signal encoded data receiving means, receiving additional data, the number V of viewpoints of the multi-view image signal and the multi-view point from the received encoded data. The first encoded data in which the correspondence between the number v for identifying each viewpoint of the image and the viewpoint ID is encoded, the decoded image output order number o, and the multi-view image signal are encoded, respectively. Second encoding Separating means for separating data, decoding the first encoded data separated, the number V of viewpoints of the multi-view image signal, and the number v and the decoded image for specifying each viewpoint of the multi-view image A first decoding means for generating supplementary additional information including a correspondence relationship with a viewpoint ID for identifying a viewpoint for management of the decoded data, decoding the separated second encoded data, and setting a decoded image output order number o Second decoding means to be generated, and a quotient (integer) obtained by dividing the decoded decoded image output order number o by the number V of decoded viewpoints by integer arithmetic, outputs decoded images at the respective viewpoints. A calculation unit that calculates the number d indicating the order, calculates a remainder of division (an integer of 0 or more and less than V) as a number v that identifies each viewpoint, and decodes the second encoded data to decode A third decoding means for generating a multi-viewpoint image signal; A storage means for storing the multi-viewpoint image signal in the image buffer, and a decoded multi-viewpoint image signal from the image buffer in accordance with the number d and the view ID indicating the output order of the decoded image at each viewpoint supplied from the calculation means And an output means for outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other. In order to achieve the above object, a multi-viewpoint image receiving program according to a sixth invention causes a computer to execute each step of the second invention.

  In the second, fourth, and sixth inventions described above, supplementary additional information indicating that the multi-view image signal is encoded, the number V of viewpoints of the multi-view image signal, and each viewpoint of the multi-view image The first encoded data in which the correspondence between the viewpoint number v that identifies each viewpoint and the viewpoint ID that identifies the viewpoint for decoding image management is encoded, the number V of viewpoints, and the number that identifies each viewpoint The second encoded data in which the decoded image output order number o collectively indicating v and the number d indicating the output order of the decoded image at each viewpoint and the multi-view image signal are respectively received are received. A viewpoint ID for specifying a viewpoint for managing the decoded image is prepared instead of the viewpoint number v for specifying the viewpoint of each image, and the viewpoint of the decoded image is managed based on the viewpoint ID Can do.

  According to the present invention, the viewpoint number v specifying the viewpoint of the image signal of each viewpoint constituting the multi-view image signal calculated from the decoded image output order number o or the viewpoint ID specifying the viewpoint and the decoding at each viewpoint. According to the number d indicating the output order of the images, the decoded image signal of each viewpoint is synchronized with each other and output in association with these parameters, so that the decoded image signal of each viewpoint can be appropriately output, The correspondence between the viewpoint and the image can be grasped by a multi-view image display device or the like that is the output destination of the decoded image.

  Furthermore, according to the present invention, when the conventional single-view image encoding / decoding method is extended to the multi-view image encoding / decoding method, the decoded image output order number o of the present invention is converted to the conventional encoding. Encoding / decoding can be realized with a small improvement by treating and decoding as a number indicating the output order of decoded images of the method (for example, picture order count in the AVC / H.264 method).

  Furthermore, according to the present invention, when the image signals of each viewpoint are output in a plurality of channels, they are sequentially arranged and output for each frame or field after decoding as in the conventional stereoscopic image receiving method and apparatus. It is not necessary to synchronize afterwards, no image buffer for synchronization is provided, and the delay time can be shortened.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, before describing the multi-view image receiving method, multi-view image receiving apparatus, and multi-view image receiving program according to the present invention, a multi-view image signal that is received and decoded according to the present invention is generated. A viewpoint image encoding device will be described.

  FIG. 1 is a block diagram showing an example of a multi-view image encoding apparatus that generates an encoded multi-view image signal to be received and decoded according to the present invention. As shown in FIG. 1, the multi-view image encoding apparatus includes an encoding management unit 101, a parameter set encoding unit 102, a multi-view image SEI encoding unit 103, a decoded image output order number calculation unit 104, a rearrangement buffer. 105, motion / disparity compensation prediction unit 106, encoding mode determination unit 107, residual signal calculation unit 108, residual signal encoding unit 109, residual signal decoding unit 110, residual signal superimposing unit 111, decoded image buffer 112 , A coded bit string generation unit 113 and a multiplexing unit 114, and the coded data (coded bit string) obtained by coding the multi-view image signal input to the multi-view image coding apparatus is transmitted by the transmitting unit. The signal is output after being converted into a predetermined signal format according to the transmission medium.

  Here, the multi-viewpoint image signal is a multi-viewpoint image signal including the image signals of the respective viewpoints obtained from a plurality of set viewpoints, and the image signal of one viewpoint is actually transmitted from the one viewpoint. It is an image signal obtained by photographing, or an image signal generated as a virtually photographed image from one viewpoint. The transmission medium includes a network, a wired transmission path, a wireless transmission path, and the like, and further includes recording on a recording medium and sending the recording medium.

  Next, the operation of the multi-view image encoding apparatus shown in FIG. 1 will be described in association with the AVC / H.264 encoding method. First, parameter information related to coding of the entire sequence and parameter information related to picture coding are supplied to the coding management unit 101 in FIG. 1. In the AVC / H.264 coding method, sequence information is converted to SPS (sequence Parameter set) and picture information are managed as PPS (picture parameter set). These pieces of information are information necessary for decoding the encoded bit string on the receiving side, and are supplied to the parameter set encoding unit 102.

  The parameter set encoding unit 102 encodes supplied sequence information, picture information, and the like. In the AVC / H.264 encoding method, an encoded bit string is configured with a NAL unit that is a delimiter of NAL (Network Abstraction Layer) as a unit, so an identifier that identifies the type of NAL unit in the header part of the NAL unit, etc. Then, sequence information, picture information, etc. are encoded. Further, the encoding management unit 101 is supplied with information on the number of viewpoints of the multi-view image supplied to the multi-view image encoding device. These pieces of information are necessary for displaying a multi-viewpoint image on the receiving side.

  In the present invention, a multi-view image SEI (supplementary additional information) indicating that the encoded bit string to be generated is a multi-view image is defined. An example of the encoding syntax structure of the payload portion (data portion) of the multi-viewpoint image SEI is shown in FIG. In FIG. 3, “num_views_minus1” is a parameter indicating the number of viewpoints of the multi-view image to be encoded. Since the number of viewpoints is 1 or more, a value obtained by subtracting 1 from the number of viewpoints is used as the AVC / H.264 encoding method. The encoding is performed using a variable length such as an exponential Golomb code prepared in the above or a fixed length that defines a length in advance.

  The multi-view image SEI encoding unit 103 in FIG. 1 encodes a multi-view image SEI (supplementary additional information) indicating that the multi-view image is encoded. That is, the multi-viewpoint image SEI encoding unit 103 first encodes an identifier or the like that identifies the type of the NAL unit in the header portion of the NAL unit, and then encodes an identifier or size that identifies the type of the payload portion of the SEI. "Num_views_minus1" is encoded based on the encoding syntax structure shown in FIG.

  Here, a value not yet defined by the AVC / H.264 encoding method is defined as an identifier value for identifying the type of the payload portion of the SEI. On the encoding side, the multi-view image SEI is encoded in this way, and on the decoding side, the encoded multi-view image SEI is decoded when the encoded bit string is decoded. It is possible to know that the content of the encoded bit string is a multi-view image and the number of viewpoints of the multi-view image. Further, even in the conventional AVC / H.264 decoder in which the multi-view image SEI is not defined, the encoded bit string can be decoded and the decoded image can be output by invalidating the multi-view image SEI. Compatibility with can be maintained. However, since the multi-view image SEI is not defined, the multi-view image SEI cannot be decoded, and whether the content of the encoded bit string is a multi-view image or not is not known. Absent.

  Further, the encoding management unit 101 is supplied with information for specifying each viewpoint, information such as a time stamp, etc. for each of the images constituting the multi-viewpoint image. Based on these information and image input order, etc., which viewpoint each image belongs to, the output order of the decoded image at each viewpoint (the encoded bit string obtained by encoding with this multi-viewpoint image encoding device) (Decoding image output order at each viewpoint obtained by decoding on the decoding side), synchronization between each viewpoint, and the like are managed. For each image, a viewpoint number v that identifies the viewpoint and a number d that indicates the output order of the decoded image at each viewpoint are assigned, and are associated with each image.

  Here, how to assign the value of the viewpoint number v that specifies each viewpoint will be described. The viewpoint number v of each viewpoint is uniquely assigned to each viewpoint by 0 or a positive integer less than V indicating the number of viewpoints. Here, the viewpoint number v for identifying each viewpoint is also used as a number indicating the output order of each viewpoint at each time on the decoding side. Therefore, in order to specify the order, the viewpoint number v at each time on the decoding side is used. According to the output order of each viewpoint, it is incremented by 1 from 0 in ascending order and assigned to the viewpoint number v of each viewpoint. Also, by assigning the same value as the viewpoint number v of the viewpoint to each image belonging to each viewpoint, it is possible to specify which viewpoint each image belongs to by the viewpoint number v.

  Next, how to assign the value of the number d indicating the output order of the decoded image from each viewpoint will be described. An integer is assigned to each number d, and the value is increased in accordance with the output order of decoded images. However, when the output times of the decoded images between the viewpoints are the same, the values of the numbers d are the same. According to this rule, when an image does not exist at a certain viewpoint even though an image exists at another viewpoint at a certain time, the value of the number d corresponding to the image at that viewpoint is missing. As described above, by assigning a value to the number d indicating the output order of each decoded image, the decoding side outputs from the image having a smaller number d according to the value of the number d, and outputs in the desired output order. In addition, when the number d is the same between the viewpoints, it can be determined that the output times are the same, so that the viewpoints can be output in synchronization. In addition, the coding management unit 101 manages the coding order of each image and manages reference images used for motion compensation prediction / parallax compensation prediction.

  FIG. 4 is a view number v that identifies the viewpoint and a number d that indicates the output order of the decoded image at each viewpoint for each of the images constituting the multi-view image encoded by the multi-view image encoding device of FIG. It is a figure which shows an example at the time of assigning a value to. In FIG. 4, the vertical axis represents the viewpoint direction, and the horizontal axis represents the time direction. M (v) (v = 0, 1, 2,..., V−1) indicates viewpoint images constituting the multi-viewpoint image, and v is a viewpoint number for specifying each viewpoint. . Further, m (v, d) (v = 0, 1, 2,..., V−1; d = 0, 1, 2,...) Represents an image constituting the viewpoint image M (v). In the figure, v is a viewpoint number that identifies each viewpoint, and d is a number that indicates the output order of the decoded image at each viewpoint. For example, when comparing two images m (v, d), if the values of the numbers v are the same, they are images of the same viewpoint, and if the values of the numbers v are different, both Are images of different viewpoints. When the values of the numbers d are the same, the images are the same time. When the values of the numbers d are different, the images are at different times, and the smaller value is the image at the earlier time.

  In addition to encoding the number of viewpoints V of the multi-viewpoint image signal, it is also possible to individually encode the viewpoint number v for specifying the viewpoint and the number d indicating the output order of the decoded image at each viewpoint. In the multi-viewpoint encoding apparatus of FIG. 1, encoding is performed as a decoded image output order number o that collectively indicates both. When the conventional single-view image encoding / decoding method is extended to the multi-view image encoding / decoding method of the present embodiment, the viewpoint number v for specifying the viewpoint of the method of FIG. The decoded image output order number o that collectively indicates the output order d of the decoded image is treated as a number that indicates the output order of the decoded image of the conventional single-view image encoding / decoding method, and is encoded / decoded. Thus, with a small improvement, compatibility with the conventional single-viewpoint image encoding / decoding scheme can be achieved. Specifically, in the AVC / H.264 system, the decoded image output order number o of this system is a number indicating the output order of decoded images of the AVC / H.264 system, and is a picture order count. Treat as.

  The decoded image output order number calculation unit 104 in FIG. 1 outputs the decoded image from the viewpoint number V of the multi-view image to be encoded, the viewpoint number v that identifies each viewpoint, and the output order d of the decoded image at each viewpoint. A sequence number o is calculated. The decoded image output order number o is calculated by the following equation.

o = d · V + v (1)
FIG. 5 calculates the decoded image output order number o according to the equation (1) for each of the images constituting the multi-view image of five viewpoints (V = 5) encoded by the multi-view image encoding device shown in FIG. FIG. 8 is a diagram illustrating an example when values are assigned. Further, FIG. 6 is a diagram for each of the images constituting a multi-view image of five viewpoints (V = 5) having different frame rates for each viewpoint encoded by the multi-view image encoding device shown in FIG. FIG. 10 is a diagram illustrating an example when a decoded image output order number o is calculated and assigned a value. However, in FIG. 6, the frame rate is different for each viewpoint, and the viewpoint images M (1) and M (3) have frame rates different from the viewpoint images M (0), M (2), and M (4). It is getting smaller. In the viewpoint images M (1) and M (3), there are no images in which the number d indicating the output order of the decoded images takes values of “1”, “3”, and “5”. There is no image in which the output order number o takes the values “6”, “8”, “16”, “18”, “26”, “28”.

  Further, in the multi-view image decoding apparatus described later on the decoding side, from the viewpoint number V of the multi-view image obtained by decoding the bit string encoded by the multi-view image encoding apparatus in FIG. 1 and the decoded image output order number o. A viewpoint number v (where v is an integer greater than or equal to 0 and less than V) and a number d (integer) indicating the output order of decoded images at each viewpoint are calculated. Specifically, the number d is a quotient obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. The number v is a remainder value obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. Alternatively, the viewpoint number v may be calculated by the following equation after calculating the number d.

v = od−V (2)
The encoded bit string generation unit 113 in FIG. 1 encodes the decoded image output order number o calculated by the decoded image output order number calculation unit 104 for each slice into a bit string together with other slice information. Here, in the AVC / H.264 encoding method, an encoded bit string is configured in units of NAL units that are a delimiter of NAL (Network Abstraction Layer), so the type of NAL unit is set in the header portion of the NAL unit. After encoding an identifier or the like for identifying, slice information including the decoded image output order number o is encoded.

  The rearrangement buffer 105 stores the supplied multi-viewpoint images. Here, the viewpoint images M (v) (v = 0, 1, 2,..., V−1) constituting the multi-viewpoint image are input in parallel through independent channels for each viewpoint, There is a method in which a viewpoint image signal is serially input through one channel as an interleaved signal. As a method of interleaving the image signals of each viewpoint, a method of interleaving the image signals of each viewpoint in units of pixels, a method of interleaving in units of a plurality of pixels, a method of interleaving in units of horizontal lines, a unit of images There are a method of interleaving, a method of interleaving in units of a plurality of images, and the like.

  An example of the interleave structure of the input viewpoint image M (v) will be described with reference to FIGS. FIG. 7 shows an example in which the signals at each viewpoint are interleaved in units of pixels. In the figure, p (v, i) represents a pixel of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and i is a pixel index.

  FIG. 8 shows an example of interleaving the signals at each viewpoint in units of a plurality of pixels. In the figure, py (v, iy) represents a pixel of the luminance signal of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and iy is an index of the pixel of the luminance signal. pu (v, iu) represents a pixel of the color difference signal (U), and iu is an index of the pixel of the color difference signal (U). pv (v, iv) represents a pixel of the color difference signal (V), and iv is an index of the pixel of the color difference signal (V).

  FIG. 9 shows an example in which the signals of each viewpoint are interleaved in units of pixel blocks of 16 pixels in the horizontal direction and 16 pixels in the vertical direction, or in units of pixel blocks of 16 pixels in the horizontal direction and 8 pixels in the vertical direction. In the figure, b (v, ib) represents a pixel of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and ib is a pixel block index. It can be said that this is a type of interleaving in a unit of a plurality of pixels by scanning a pixel block along a horizontal line.

  FIG. 10 shows an example of interleaving the signals of each viewpoint in units of horizontal lines. In the figure, l (v, j) represents a line of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and j is an index of the line. Since a line is composed of a plurality of pixels, it can be said that the line is a kind of interleaved unit of a plurality of pixels.

  FIG. 11 shows an example of interleaving the signals of the respective viewpoints in a form in which the signals are combined into one image. In this case, it can be said that this is a type of interleaving in units of horizontal lines by scanning a single image with horizontal lines. FIG. 11A shows one image in which the signals of the respective viewpoints are collected, and FIG. 11B shows an image interleaved in units of horizontal lines. In the same figure, v indicates a viewpoint number for specifying a viewpoint, and d indicates a number indicating an imaging order of images at each viewpoint (a number indicating an output order of decoded images at each viewpoint). Further, j in l (v, j) indicates a line index.

  FIG. 12 shows an example in which the signals of each viewpoint are interleaved in units of slices in which a plurality of lines are combined. In the figure, s (v, k) represents a pixel of a luminance signal of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and k is an index of a slice in which a plurality of lines are collected. Since a slice is composed of a plurality of pixels, it can be said that it is a kind of interleaving in units of a plurality of pixels and also a kind of interleaving in units of pixel blocks.

  FIG. 13 shows an example in which the signals of each viewpoint are interleaved in units of images. In the figure, m (v, d) represents an image constituting each viewpoint image, v is a viewpoint number for specifying the viewpoint, and d is a number indicating an image capturing order at each viewpoint (at each viewpoint). The number indicating the output order of the decoded image). The image signals of the respective viewpoints having the same number d are set as one group, and in that group, the values of the number v are sequentially input in order from the smallest. Furthermore, by inputting in order from the group with the smallest value of the number d, the viewpoints can be input in synchronization with each other in units of images.

  FIG. 14 shows an example in which the signals of each viewpoint are interleaved in units of a plurality of images. In the same figure, m (v, d) represents an image constituting each viewpoint image as in FIG. 13, v is a viewpoint number for specifying the viewpoint, and d is a number indicating the image capturing order at each viewpoint ( The number indicating the output order of the decoded image at each viewpoint).

  In the multi-view image encoding device of FIG. 1, even when an image signal is input by any of the methods in FIGS. 7 to 14, the image signal is appropriately input to the rearrangement buffer 105 and stored without delay. Further, according to the encoding order controlled by the encoding management unit 101, the image signal stored in the rearrangement buffer 105 is appropriately supplied to the motion / disparity compensation prediction unit 106 and the residual signal calculation unit 108 in units of pixel blocks, respectively. Supplied.

  As described above, in the multi-view image encoding apparatus shown in FIG. 1, a method of inputting the viewpoint images of the respective viewpoints in parallel through independent channels, and the viewpoint images of the respective viewpoints as an interleaved signal 1 In any of the serial input methods using one channel, the viewpoint images of the respective viewpoints are input and stored in the rearrangement buffer 105 as appropriate without delay. Therefore, unlike the conventional stereoscopic image encoding method and apparatus, it is not necessary to sequentially arrange one frame or one field before encoding, and there is no image buffer for sequentialization, and the delay time is shortened. The effect that it can be done can be obtained.

  Next, the encoding order controlled by the encoding management unit 101 will be described with reference to FIG. FIG. 15 is a diagram illustrating an example of the encoding order of each image m (v, d) constituting a multi-view image of five viewpoints (V = 5) and a reference relationship of motion compensation / parallax compensation. In the figure, viewpoint images M (0), M (2), and M (4) are encoded without performing disparity compensation prediction referring to images of other viewpoints. For example, the image m (0,0) of the viewpoint image M (0) is encoded as a picture that is encoded independently only within the screen without referring to other images. Further, the image m (0,3) of the viewpoint image M (0) is encoded using the motion compensated prediction with the decoded image of the front image m (0,0) in the display order of the same viewpoint as the reference image. . Further, the image m (0, 1) of the viewpoint image M (0) is a reference image that is a decoded image of the front image m (0, 0) and the rear image m (0, 3) in the same viewpoint display order. Encode using motion compensated prediction.

  On the other hand, viewpoint images M (1) and M (3) are encoded using disparity compensation prediction that predicts an image of another viewpoint as a reference image in addition to motion compensation prediction. For example, the image m (1,1) of the viewpoint image M (1) is a reference image that is a decoded image of the front image m (1,0) and the rear image m (1,3) in the display order of the same viewpoint. In addition to performing motion compensation prediction, the decoded images of the images m (0, 1) and m (2, 1) of different viewpoints are used as reference images and encoded using parallax compensation prediction.

  When the image m (1, 1) of the viewpoint image M (1) is encoded, the images m (1, 0), m (1, 3), m (0, 1), and m ( 2 and 1) must be encoded and decoded and stored in the decoded image buffer 112. In the reference relationship of this example, m (0,0), m (2,0), m (1,0), m (4,0), m (3,0), m (0,3), m (2,3), m (1,3), m (4,3), m (3,3), m (0,1), m (2,1), m (1,1), m ( 4,1), m (3,1), m (0,2), m (2,2), m (1,2), m (4,2), m (3,2), m (0 , 6), m (2, 6), m (1, 6), m (4, 6), m (3, 6), m (0, 4), m (2, 4), m (1, 4) Encoding is performed in the order of encoding. This encoding order has a one-to-one correspondence with the decoded image output order number o and is managed by the encoding management unit 101. Here, in this example, the encoding order of the images in the viewpoint direction is the same at any time, and the encoding order of the images in the time direction is the same at any viewpoint.

  The motion / disparity compensation prediction unit 106 performs the above-described disparity compensation prediction in addition to performing the motion compensation prediction in the same manner as in the conventional AVC / H.264 scheme or the like. In motion compensated prediction, an image of the same viewpoint in front or rear in the display order is used as a reference image, but in parallax compensated prediction, a common process can be performed if an image of another viewpoint is used as a reference image. In accordance with control of the encoding management unit 101, block matching is performed between the pixel block supplied from the rearrangement buffer 105 and the reference image supplied from the decoded image buffer 112. In the case of motion compensation prediction, a motion vector is used. In the case of disparity compensation prediction, a disparity vector is detected, a motion compensation prediction / disparity compensation prediction block signal is generated, and the motion compensation prediction / disparity compensation prediction block signal and the motion vector / disparity vector are input to the encoding mode determination unit 107. Supply.

  The encoding management unit 101 controls candidate combinations such as whether to perform motion compensation prediction / disparity compensation prediction, the number of reference images, which decoded image to use as a reference image, and the size of a pixel block. Accordingly, motion compensation prediction / disparity compensation prediction is performed for all combinations that are candidates for all coding modes related to motion compensation prediction / disparity compensation prediction, and each motion compensation prediction / disparity compensation prediction block signal and motion vector / disparity are predicted. The vector is supplied to the encoding mode determination unit 107. The candidate pixel block size here is each small block obtained by further dividing the pixel block. For example, if the pixel block is 16 pixels in the horizontal direction and 16 pixels in the vertical direction (that is, 16 × 16), 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, 4 × 4, etc. Dividing into small blocks, motion compensation prediction is performed, and candidates are set.

  The encoding mode determination unit 107 determines which method of motion compensation prediction and parallax compensation prediction is selected and combined in which pixel block unit using which reference image and efficient encoding can be realized. Encoding mode and the obtained encoding mode and the motion vector / disparity vector obtained are supplied to the encoded bit string generation unit 113, and the motion compensation prediction / disparity compensation prediction block signal is supplied to the residual signal calculation unit 108. Supply.

  For example, when combining motion compensated prediction from the previous and subsequent reference images on the time axis, the motion compensated prediction block obtained by performing the motion compensated prediction from the previous reference image and the motion compensated prediction from the subsequent reference image are used. A block obtained by averaging the pixel values of the motion compensated prediction block obtained by the above is generated and set as a candidate. Also, motion compensation prediction and parallax compensation prediction can be combined. Furthermore, when averaging pixel values, weighting such as 1: 2 or 1: 3 may be used in addition to the average of 1: 1. Further, when the pixel block is divided into small blocks of 4 × 4 pixels to 16 × 16 pixels to be candidates for the encoding mode, the prediction method of each small block can be changed.

  There are various methods for determining the encoding mode. For example, there is a method for calculating the code amount and the distortion amount for each encoding mode and selecting the optimum encoding mode in the balance between the code amount and the distortion amount. is there. In this encoding mode determination, first, a residual signal is calculated for each combination of encoding modes, and the bit length of an encoded sequence obtained by encoding the residual signal, vector, and encoding mode is calculated. Code amount. Further, the encoded residual signal is decoded, and an absolute value error sum or a square sum of the decoded signal added with the prediction signal and the image signal before encoding is calculated to obtain the distortion amount. The code amount is multiplied by a predetermined multiplier and added to the distortion amount to obtain an evaluation value. The smallest evaluation value of the combinations of all candidate encoding modes is selected and set as the encoding mode of the pixel block.

  The residual signal calculation unit 108 subtracts the determined motion compensation prediction / disparity compensation prediction block signal supplied from the coding mode determination unit 107 from the pixel block signal supplied from the rearrangement buffer 105 to obtain a residual signal. Get. The residual signal encoding unit 109 performs residual signal encoding processing such as orthogonal transformation and quantization on the residual signal input from the residual signal calculation unit 108, and calculates an encoded residual signal.

  The encoding management unit 101 manages whether the encoded image is used as a reference image for motion compensation prediction of an image that follows in the encoding order, or a parallax compensation prediction of another viewpoint. When the encoded residual signal is used, the encoded residual signal is decoded, and the decoded image signal is sequentially stored in the decoded image buffer 112 in units of pixel blocks.

  First, the residual signal decoding unit 110 performs residual signal decoding processing such as inverse quantization and inverse orthogonal transformation on the encoded residual signal input from the residual signal encoding unit 109, and thereby obtains a decoded residual. Generate a signal. The residual signal superimposing unit 111 superimposes the decoded residual signal supplied from the residual signal decoding unit 110 on the determined motion compensation prediction / disparity compensation prediction block signal supplied from the coding mode determination unit 107, A decoded image signal is generated, and the decoded image signal is sequentially stored in the decoded image buffer 112 in units of pixel blocks. The decoded image signal stored in the decoded image buffer 112 becomes a reference image for motion compensated prediction of an image that follows in the coding order or parallax compensated prediction of another viewpoint, if necessary.

  The encoded bit string generation unit 113, following the slice information including the decoded image output order number o described above, the determined encoding mode input from the encoding mode determination unit 107, the motion vector or the disparity vector, and the residual An encoded residual signal or the like input from the signal encoding unit 109 is sequentially encoded using entropy encoding such as Huffman encoding or arithmetic encoding to generate an encoded bit string.

  The above-described encoding process in units of pixel blocks is repeated until encoding of all pixel blocks in the encoded image is completed. Further, the encoding process in units of encoded images is repeated until all encoding is completed.

  Encoded bit sequence of parameter information related to encoding of entire sequence encoded by parameter set encoding unit 102, encoded bit sequence of parameter information related to encoding of picture, and encoded by multi-view image SEI encoding unit 103 Slice information, coding mode, motion vector or disparity vector, and coding residual including the coded bit sequence of the multi-view image SEI that has been performed and the decoded image output sequence number o coded by the coded bit sequence generation unit 113 An encoded bit string such as a signal is supplied to the multiplexing unit 114 and is multiplexed into one encoded bit string as necessary.

  At the time of multiplexing, packets are packetized based on standards such as MPEG-2 system, MP4 file format, RTP, etc. as necessary, and multiplexed by adding a packet header. The encoded bit sequence (encoded data) multiplexed into one encoded bit sequence by the multi-view image encoding device in FIG. 1 is transmitted to the receiving side via a network or the like by a transmission unit (not shown). Here, in a system or the like in which a highly reliable channel is prepared for a NAL unit including important information such as a parameter set in addition to a channel that transmits the VCL NAL unit, a plurality of codes corresponding to the channel are provided. Multiplexed into a generalized bit string. Further, this embodiment can be used not only for network transmission but also for recording on a storage medium such as a DVD, broadcasting such as BS / terrestrial waves, and cable broadcasting.

  Next, a multi-view image encoding processing procedure by the multi-view image encoding device shown in FIG. 1 will be described with reference to the flowchart of FIG. Since the processing operation of each step is the same as that described with reference to the block diagram of FIG. 1, only the processing procedure will be described here in association with FIG.

  First, in step S101, parameter information relating to encoding of the entire sequence, parameter information relating to encoding of the picture, and the like are encoded, an encoded bit string of parameter information relating to encoding of the entire sequence, and parameters relating to encoding of the picture. An encoded bit string of information is generated. The processing in step S101 corresponds to the encoding operation in the parameter set encoding unit 102 in the multi-view image encoding apparatus in FIG.

  Subsequently, in step S102, an encoded bit string of parameter information related to encoding of the entire sequence, an encoded bit string of parameter information related to encoding of a picture, and the like are multiplexed to obtain a multiplexed encoded bit string. The processing in step S102 corresponds to the multiplexing operation in the multiplexing unit 114 in the multi-view image encoding device in FIG.

  In the next step S103, a multi-view image SEI (supplementary additional information) indicating that the multi-view image is encoded is encoded, and an encoded bit string of the multi-view image SEI is generated. The processing in step S103 corresponds to the encoding operation in the multi-view image SEI encoding unit 103 in the multi-view image encoding device of FIG. Subsequently, in step S104, the multi-view image SEI is multiplexed subsequent to the encoded bit string multiplexed in step S102. The processing in step S104 corresponds to the multiplexing operation in the multiplexing unit 114 in the multi-view image encoding device in FIG.

  Subsequently, in step S105, the decoded image output order number o is calculated from the viewpoint number V of the multi-view image to be encoded, the viewpoint number v that identifies each viewpoint, and the output order d of the decoded image at each viewpoint. To do. The processing in step S105 corresponds to a calculation operation in the decoded image output order number calculation unit 104 in the multi-viewpoint image encoding device in FIG.

  In the next step S106, the decoded image output order number o is encoded to generate an encoded bit string. The processing in step S106 corresponds to the encoding operation of the decoded image output order number o in the encoded bit string generation unit 113 in the multi-view image encoding device in FIG. Subsequently, in step S107, motion compensation prediction / disparity compensation prediction is performed. The processing in step S107 corresponds to the processing operation in the motion / disparity compensation prediction unit 106 in the multi-viewpoint image encoding device in FIG.

  Subsequently, in step S108, the encoding mode is determined. The processing in step S108 corresponds to the processing operation in the encoding mode determination unit 107 in the multi-view image encoding device in FIG. In the next step S109, the motion compensation prediction / disparity compensation prediction block signal determined in step S108 is subtracted from the pixel block signal to be encoded to obtain a residual signal. The processing in step S109 corresponds to the processing operation in the residual signal calculation unit 108 in the multi-view image encoding device in FIG.

  Subsequently, in step S110, residual signal encoding processing such as orthogonal transformation and quantization is performed on the residual signal to calculate an encoded residual signal. The processing in step S110 corresponds to the processing operation in the residual signal encoding unit 109 in the multi-view image encoding device in FIG. In the next step S111, it is determined whether the encoded image is used as a reference image for motion compensation prediction of an image that follows in the encoding order or parallax compensation prediction of another viewpoint. The processing in step S111 corresponds to a determination operation as to whether or not to be used as a reference image in the encoding management unit 101 in the multi-view image encoding device in FIG. If it is determined that the image is used as a reference image, the process proceeds to step S112. If it is determined that the image is not used as a reference image, the process proceeds to step S115.

  Subsequently, in step S112, the encoded residual signal is subjected to residual signal decoding processing such as inverse quantization and inverse orthogonal transform to generate a decoded residual signal. The processing in step S112 corresponds to the processing operation in the residual signal decoding unit 110 in the multi-view image encoding device in FIG. Subsequently, in step S113, the decoded residual signal is superimposed on the determined motion compensation prediction / disparity compensation prediction block signal to generate a decoded image signal. The processing in step S113 corresponds to the processing operation in the residual signal superimposing unit 111 in the multi-view image encoding device in FIG. The decoded image signal is stored in the decoded image buffer in units of pixel blocks as appropriate.

  Subsequently, in step S114, the encoded bit sequence of the decoded image output order number o in step S106, the encoding mode determined in step S108, and the motion vector or disparity vector, the encoding residual obtained in step S108. A signal or the like is encoded to generate an encoded bit string. The processing in step S114 corresponds to the processing operation in the residual signal superimposing unit 111 in the multi-view image encoding device in FIG.

  Subsequently, in step S115, it is determined whether or not the encoding process has been completed for all the pixel blocks in the encoded image. If completed, the process proceeds to step S116. If not completed, the process proceeds to step S107, and the processes from step S107 to step S114 are repeated until the encoding process is completed for all the pixel blocks in the encoded image.

  Subsequently, in step S116, following the bit sequence multiplexed in step S102 and step S104, the decoded image output order number o, the encoding mode, and the motion vector or disparity vector, encoding residual signal, and the like are encoded. The bit string is appropriately multiplexed into one encoded bit string or a plurality of encoded bit strings as necessary. The processing in step S116 corresponds to the multiplexing operation in the multiplexing unit 114 in the multi-view image encoding device in FIG.

  Next, a multiplexing and transmission processing procedure in the multiplexing unit 114 when transmitting via a network will be described with reference to FIG. In step S301, packetization is performed based on standards such as MPEG-2 system, MP4 file format, RTP, and the like as necessary. In step S302, a packet header is added according to standards such as MPEG-2 system, MP4 file format, RTP, etc. as necessary. In step S303, transmission is performed via the network.

  Returning again to FIG. In step S117, it is determined whether or not the encoding process has been completed for all images of the multi-viewpoint image to be encoded. If completed, the multi-viewpoint image encoding processing procedure ends. If not completed, the process proceeds to step S105, and the process from step S105 to step S106 is repeated until the encoding process is completed for all the images of the multi-viewpoint image to be encoded.

  As described above, according to the transmission apparatus including the multi-viewpoint image encoding apparatus of FIG. 1, the viewpoint number v that specifies the viewpoint of the image signal of each viewpoint that constitutes the multi-viewpoint image signal and the decoded image at each viewpoint. A decoded image output order number o that collectively indicates an output order d is encoded and transmitted together with the number of viewpoints V of the multi-view image signal, and is encoded in this way on the receiving side described below By receiving and decoding the data, the multi-viewpoint image display device and the like can be appropriately output according to the viewpoint number v and the number d indicating the output order of the decoded image at each viewpoint.

  Next, the multi-view image receiving method, multi-view image receiving apparatus, and multi-view image receiving method according to the present invention for receiving and decoding the encoded data transmitted by the transmitting apparatus including the multi-view image encoding apparatus of FIG. The program will be described with reference to the drawings. FIG. 17 is a block diagram showing an embodiment of a multi-view image decoding apparatus as a main part of the multi-view image receiving apparatus according to the present invention.

  As shown in FIG. 17, the multi-view image decoding apparatus includes a separation unit 201, a parameter set decoding unit 202, a multi-view image SEI decoding unit 203, an encoded bit string decoding unit 204, a decoded image management information calculation unit 205, a motion / A parallax compensation prediction unit 206, a prediction signal synthesis unit 207, a residual signal decoding unit 208, a residual signal superimposing unit 209, a decoded image buffer 210, a decoded image management unit 211, and a decoded image output unit 212 are provided. An encoded bit string obtained by encoding a multi-view image signal received and demodulated as necessary is input, and is decoded to output a multi-view image signal.

  Next, the operation of the multi-view image decoding apparatus shown in FIG. 17, which is the main part of the receiving apparatus of the present invention, will be described in association with the AVC / H.264 encoding method. First, the separation unit 201 receives an encoded bit string that has been encoded by the multi-view image encoding apparatus illustrated in FIG. 1 and transmitted via the network. It should be noted that the encoded bit string supply form in this system is not only received via network transmission, but also a code bit string recorded on a storage medium such as a DVD, or a code broadcast on BS / terrestrial broadcasts. An encoded bit string can also be received.

  Further, the separation unit 201 removes the packet header from the supplied encoded bit string, and separates the NAL unit. Further, an identifier (nal_unit_type) for identifying the type of the NAL unit included in the header portion of the separated NAL unit is evaluated, parameter information related to coding of the entire sequence, parameter information related to picture coding, etc. Is encoded bit sequence encoded in the multi-view image SEI decoding unit 203, the NAL unit is an encoded bit sequence in which the multi-view image SEI is encoded. When the NAL unit is a VCL NAL unit, that is, a decoded image output order number o, a coding mode, and a coded bit string in which a motion / disparity vector, a coded residual signal, and the like are coded, This is supplied to the encoded bit string decoding unit 204. However, when discriminating whether or not the separated NAL unit is a multi-viewpoint image SEI, an identifier that distinguishes the type of the NAL unit is judged as SEI, and then an identifier that distinguishes the type of the payload portion of the SEI is evaluated. Indicates that it is a multi-view image SEI.

  Next, the parameter set decoding unit 202 decodes the encoded bit string in which the parameter information related to the encoding of the entire sequence separated by the separating unit 201, the parameter information related to the encoding of the picture, and the like are encoded, and the entire sequence Parameter information related to the encoding of the image, parameter information related to the encoding of the picture, etc. (These parameter information is used in the processing of the entire multi-viewpoint image decoding apparatus, but is not used in the description of the present invention. Not shown in the drawing).

  The multi-view image SEI decoding unit 203 decodes the encoded bit string obtained by encoding the multi-view image SEI separated by the separation unit 201 based on the syntax structure described with reference to FIG. Information indicating that the content of the encoded coded bit string supplied to the decoding device is a multi-view image and information on the number of viewpoints are obtained. Here, since “num_views_minus1” in FIG. 4 is encoded by a value obtained by subtracting “1” from the number of viewpoints V, the value obtained by adding “1” to the value of “num_views_minus1” is set as the number of viewpoints V.

  Here, when the multi-view image SEI is not included in the encoded bit sequence supplied to the multi-view image decoding apparatus, it is determined that the content of the encoded bit sequence is not a multi-view image, and the conventional single-view AVC / H It is regarded as an encoded bit string encoded by the .264 encoding method, and decoding is performed with the number of viewpoints set to “1”.

  The encoded bit string decoding unit 204 encodes the decoded image output order number o, the encoding mode, the motion / disparity vector, the encoded residual signal (encoded prediction residual signal), and the like separated by the separating unit 201. The encoded bit sequence is decoded to obtain a decoded image output order number o, an encoding mode, a motion / disparity vector, an encoded residual signal, and the like.

  The decoded image management information calculation unit 205 uses the viewpoint number V supplied from the multi-view image SEI decoding unit 203 and the decoded image output order number o supplied from the encoded bit string decoding unit 204 to satisfy each image satisfying Expression (1). The viewpoint number v (where v is an integer between 0 and less than V) and the number d (integer) indicating the output order of the decoded image at each viewpoint are calculated. Specifically, the number d is a quotient obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. The number v is a remainder value obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. Alternatively, the viewpoint number v may be calculated by the equation (2) after calculating the number d.

  In this way, the viewpoint number v for specifying the viewpoint of each image and the number d indicating the output order of the decoded image at each viewpoint are calculated from the number of viewpoints V and the decoded image output order number o. 5. As shown in FIG. 6, not only when a value is assigned to the decoded image output order number o, but also when a packet is lost due to an error during transmission through the network, which image of which viewpoint is normally It is possible to identify whether it is missing.

  The viewpoint number v specifying the viewpoint calculated by the decoded image management information calculation unit 205 and the number d indicating the output order of the decoded image at each viewpoint are the number of viewpoints V supplied from the multi-viewpoint image SEI decoding unit 203, Along with the decoded image output order number o supplied from the encoded bit string decoding unit 204, the decoded image output order number o is supplied to a decoded image management unit 211, which will be described later, and used for management of the decoded image stored in the decoded image buffer 210.

  Here, in the multi-view image decoding apparatus, when the conventional single-view decoding method is expanded as a multi-view decoding method, as in the case of encoding, the decoded image output order number o of the present method is set to the conventional single-view decoding method. By treating it as a number indicating the output order of the decoded image of the decoding method of one viewpoint, compatibility with the conventional decoding method can be achieved.

  For example, when the AVC / H.264 system is expanded to the multi-viewpoint encoding system, the decoded image output order number o of this system is a picture / number indicating the output order of the decoded image of the AVC / H.264 system. Treat as an order count. In addition, since the multi-view image decoding apparatus performs decoding in the order encoded on the encoding side, the encoding order is equal to the decoding order. Further, the encoded bit string decoding unit 204 obtains information such as an encoding mode of a pixel block to be decoded, a motion vector or a disparity vector, an encoded residual signal (encoded prediction residual signal), and the like.

  Subsequently, the motion / disparity compensation prediction unit 206 performs motion compensation prediction / disparity compensation prediction according to the coding mode decoded by the coded bit string decoding unit 204 and the motion vector / disparity vector. In this motion compensation prediction / disparity compensation prediction, a pixel block at a position indicated by the motion vector / disparity vector decoded by the encoded bit string decoding unit 204 with reference to an image supplied from the decoded image buffer 210 according to the encoding mode. Is a motion compensation prediction / parallax compensation prediction block. The size of the pixel block is divided into small blocks, and the prediction method and motion vector / disparity vector of each small block may be different. In some cases, prediction is performed from a plurality of reference pictures. In such a case, a plurality of motion compensation predictions / disparity compensation predictions are performed to obtain a plurality of prediction blocks.

  The prediction signal synthesis unit 207 synthesizes a plurality of prediction blocks when the pixel block is divided into small blocks or is predicted from a plurality of reference pictures, and generates a prediction signal of the pixel block. On the other hand, the residual signal decoding unit 208 performs a residual signal decoding process such as inverse quantization and inverse orthogonal transform on the encoded residual signal input from the encoded bit string decoding unit 204, thereby obtaining a decoded residual signal. Is generated.

  The residual signal superimposing unit 209 calculates a decoded image signal by superimposing the decoded residual signal supplied from the residual signal decoding unit 208 on the prediction signal supplied from the prediction signal combining unit 207, and outputs a decoded image buffer 210. The decoded image signals are sequentially stored in pixel block units. The decoded image signal stored in the decoded image buffer 210 serves as a reference image when decoding subsequent images in the encoding order as necessary.

  The above-described decoding process for each pixel block is repeated until the decoding of all the pixel blocks in the decoded image is completed for each pixel block.

  The decoded image management unit 211 outputs the number V of viewpoints supplied from the decoded image management information calculation unit 205, the decoded image output order number o, the viewpoint number v that identifies the viewpoint of each image, and the output of the decoded image at each viewpoint. The number d indicating the order and the decoded image signal stored in the decoded image buffer 210 are managed in association with each other. The decoded image management unit 211 determines whether or not to output the decoded image stored in the decoded image buffer 210 based on these parameters.

  When the decoding order and the output order of the decoded image are different and the output timing is different, a delay is necessary, and the decoded image may not be output. The decoded image management unit 211 identifies viewpoints for each decoded image signal stored in the decoded image buffer 210 by a number v, manages the output order of the decoded images at each viewpoint by a number d, and manages each viewpoint. Control is performed so that the viewpoints are synchronized with each other so that images with the same value of the number d of the decoded image signals are output simultaneously or continuously, and output in ascending order of the value of the number d. The decoded image output unit 212 outputs the decoded image stored in the decoded image buffer 210 to the multi-view image display device or the like in synchronization with the decoded image signals of the respective viewpoints according to the control of the decoded image management unit 211. Here, the viewpoint number v that identifies the viewpoint of each image and the number d that indicates the output order of the decoded image at each viewpoint are associated with the decoded image signal of each viewpoint, and output together with these parameters as necessary. .

  As a method of outputting the images of each viewpoint of the decoded viewpoint image M (v) in synchronization with each other, a method of outputting the image signals of each viewpoint in parallel on independent channels, and an interleaving of the image signals of each viewpoint There is a method of serially outputting with one channel. When the decoded image output unit 212 outputs the image signals of the respective viewpoints in parallel on independent channels, the number d indicating the output order of the decoded images at the respective viewpoints is ordered in ascending order. The image signals at the same time, that is, the decoded image signals of the respective viewpoints having the same number d indicating the output order of the decoded images at the respective viewpoints are output in synchronization with each other.

  If it demonstrates using FIG. 4 used by description of the above-mentioned multiview image coding apparatus, for each viewpoint image M (v), the value of the number d of each image m (v, d) is in order from the smallest. By outputting the decoded images, the decoded images can be output in a desirable order when displayed on an output destination multi-viewpoint image display device or the like. Further, by outputting the decoded image signals of the respective viewpoints having the same value of the number d at the same time, the decoded image signals of the respective viewpoints can be synchronized with each other. At that time, by starting output of the decoded image signal after the image signals of all the viewpoints are decoded, the decoded image signals of the respective viewpoints can be output without being lost.

  When the decoded image output unit 212 serially outputs a signal as an interleaved view of each viewpoint using one channel, the number d indicating the output order of the decoded image at each viewpoint is counted in ascending order. Images at the same time, that is, the decoded image signals of the respective viewpoints having the same number d indicating the output order of the decoded images at the respective viewpoints, are synchronized with each other by interleaving. As a method of serially outputting each viewpoint as an interleaved signal, a method of interleaving each viewpoint signal in units of pixels, a method of interleaving in units of a plurality of pixels, a method of interleaving in units of horizontal lines, There are a method of interleaving in units of images, a method of interleaving in units of a plurality of images, and the like.

  The interleave structure of the decoded image signal to be output is the same as the interleave structure of the viewpoint image input to the above-described multi-view image encoding device shown in FIGS. As in the case of outputting in parallel on independent channels, the method of interleaving each viewpoint signal in units of pixels, the method of interleaving in units of a plurality of pixels, and the method of interleaving in units of horizontal lines, for example, As shown in FIG. 7 to FIG. 12, each image signal of each viewpoint having the same number d is assigned to a plurality of pixels in order from the smallest number d indicating the output order of the decoded image at each viewpoint. Are interleaved and output in units of horizontal lines and output in units of horizontal lines, so that the image signals of the viewpoints can be output at the same time, and the viewpoints can be output in synchronization with each other. At this time, by starting the interleaving and output of the decoded image signal after the image signals of all viewpoints to be interleaved are decoded, the decoded image signals of the respective viewpoints can be output without being lost.

  Further, in the method of interleaving in units of images, as shown in FIG. 13, the image signals of the viewpoints having the same number d are grouped into one group, and the value of the number v is continuously small in units of images in that group. Output in order. Furthermore, by outputting in order from the group having the smallest value of the number d, the viewpoints can be output in synchronization with each other in units of images.

  As described above, in the multi-view image decoding apparatus shown in FIG. 17, when outputting the decoded image stored in the decoded image buffer 210 to the multi-view image display apparatus or the like, it matches the input of the output destination display apparatus or the like. The output is performed in a format, that is, a method in which each viewpoint is output in parallel on independent channels, or a method in which each viewpoint is serially output on one channel as an interleaved signal. Therefore, unlike the conventional stereoscopic image decoding method and apparatus, there is no need to synchronize after decoding by sequentially arranging every frame or field after decoding, and there is no image buffer for synchronization. The effect that the delay time can be shortened can be obtained.

  The above decoding process is repeated until all the decoding processes of the encoded bit string to be decoded are completed.

  In the above description, in the decoded image management unit 211, the number V of viewpoints supplied from the decoded image management information calculation unit 205, the decoded image output order number o, the viewpoint number v specifying the viewpoint of each image, and each In the above description, the number d indicating the output order of the decoded image from the viewpoint of the image and the decoded image signal stored in the decoded image buffer 210 are managed in association with each other. However, the viewpoint that specifies the viewpoint of each image obtained by calculation Instead of the number v, a viewpoint ID for specifying a viewpoint can be prepared for managing the decoded image, and the viewpoint of the decoded image can be managed or the decoded image can be output based on the viewpoint ID.

  However, the viewpoint number v that specifies the viewpoint of each image obtained by calculation and the viewpoint ID that specifies the viewpoint for managing the decoded image need to correspond one-to-one. That is, both may be the same value or different values. The viewpoint ID for specifying the viewpoint for management of the decoded image may be generated by the decoding device as necessary, or the viewpoint ID for specifying the viewpoint for management of the decoded image is encoded on the encoding side and the value is set. It may be used. When encoding the viewpoint ID for identifying the viewpoint for managing the decoded image on the encoding side, the viewpoint ID for identifying the viewpoint for managing the decoded image by the encoding management unit 101 in FIG. A one-to-one correspondence relationship with the specified viewpoint number v is managed, and the viewpoint ID is encoded so that the multi-view image SEI encoding unit 103 can determine the one-to-one correspondence relationship on the decoding side.

  In this case, an example of the encoding syntax structure of the multi-view image SEI encoded by the multi-view image SEI encoding unit 103 is shown in FIG. In FIG. 19A, “num_views_minus1” is a parameter indicating the number of viewpoints of a multi-view image to be encoded in the same manner as described in FIG. 3, and the number of viewpoints is “1” or more. A value obtained by subtracting 1 ″ is encoded with a variable length such as an exponential Golomb code prepared in the AVC / H.264 encoding method or a fixed length with a predetermined length. View_id [v] indicates a viewpoint ID for specifying a viewpoint for management of a decoded image of the viewpoint indicated by the viewpoint number v specifying each viewpoint, and is an index prepared in the AVC / H.264 encoding method. Encode with variable length such as Exponential Golomb code or fixed length with pre-defined length. The for loop indicates that the viewpoint IDs are encoded in ascending order from the viewpoint having the viewpoint number v that specifies the viewpoint of each image having a value of 0.

  On the decoding side, the multi-view image SEI decoding unit 203 in FIG. 17 performs decoding according to the syntax structure shown in FIG. 19A, and in addition to the number of viewpoints V of the multi-view image, a viewpoint number v that identifies each viewpoint. View_id [v], which is a viewpoint ID for specifying the viewpoint, is managed for management of the decoded image of the viewpoint indicated by. The one-to-one correspondence between the viewpoint number v that identifies each viewpoint and the viewpoint ID is managed by the decoded image management unit 211 through the decoded image management information calculation unit 205. In addition, the decoded image output unit 212 associates the viewpoint ID with the decoded image signal of each viewpoint, instead of the viewpoint number v that specifies the viewpoint of each image, as necessary, and outputs them simultaneously.

  Further, in the description using FIG. 19A, the ID is encoded in ascending order from the viewpoint having the viewpoint number v 0 that specifies the viewpoint of each image, but the encoding of each viewpoint at the same time is described. The viewpoint ID can be encoded in the decoding order. In this case, an example of an encoding syntax structure of the multi-view image SEI encoded by the multi-view image SEI encoding unit 103 is shown in FIG. i indicates the encoding / decoding order of each viewpoint at the same time, and i of each viewpoint has a value that increases one by one in ascending order from 0 according to the encoding / decoding order. view_id [i] indicates a viewpoint ID for identifying a viewpoint for management of a decoded image of the viewpoint indicated by the number i indicating the encoding / decoding order of each viewpoint at the same time, and is prepared in the AVC / H.264 encoding scheme. It is encoded with a variable length such as an exponential Golomb code or a fixed length with a predetermined length. The for loop indicates that viewpoint IDs are encoded in ascending order from a viewpoint having a value of number i indicating the encoding / decoding order of each viewpoint at the same time.

  On the decoding side, the multi-view image SEI decoding unit 203 in FIG. 17 performs decoding according to the syntax structure shown in FIG. 19B, and encodes / decodes each viewpoint at the same time in addition to the number of viewpoints V of the multi-view image. View_id [i], which is a viewpoint ID for specifying the viewpoint, is decoded for management of the decoded image of the viewpoint indicated by the number i indicating the order. The viewpoint ID and the number i indicating the encoding / decoding order of each viewpoint at the same time have a one-to-one correspondence, and the encoding / decoding order of each viewpoint at the same time as the viewpoint number v that identifies each viewpoint. Therefore, the viewpoint number v for identifying each viewpoint and the viewpoint ID also have a one-to-one correspondence. These one-to-one correspondences are managed by the decoded image management unit 211 through the decoded image management information calculation unit 205. In addition, the decoded image output unit 212 associates the viewpoint ID with the decoded image signal of each viewpoint, instead of the viewpoint number v that specifies the viewpoint of each image, as necessary, and outputs them simultaneously.

  Next, the multi-view image decoding processing procedure by the multi-view image receiving method according to the present invention will be described with reference to the flowchart of FIG. Since the processing operation of each step is the same as that described with reference to the block diagram of FIG. 17, only the processing procedure will be described here in association with FIG.

  First, in step S201, the received encoded bit string is separated into NAL unit units. In step S201, the reception and separation processing procedure shown in FIG. 20 is performed in the reception apparatus that receives the multi-view image encoded data transmitted via the network. In step S401 in FIG. 20, a coded bit string is received via a network by a receiving unit (not shown in FIG. 17). In step S402, the packet header added based on the MPEG-2 system method, MP4 file format, RTP, etc. used in the encoded bit string received by the receiving unit is decoded and removed. The received encoded bit string is separated in units of NAL units by the separation unit 201 in FIG. 17 (step S403).

  Returning to FIG. The identifier (nal_unit_type) for identifying the type of the NAL unit included in the header part of the NAL unit separated in step S201 in FIG. 18 is evaluated, and the NAL unit is used to encode the parameter information and the picture related to the coding of the entire sequence. It is determined whether or not it is a parameter set such as related parameter information (step S202). If it is a parameter set, the process proceeds to step S206, and if it is determined not to be a parameter set but SEI (step S203), the process proceeds to step S205. In step S205, an identifier for identifying the type of the payload portion of the SEI is evaluated, and in the case of the multi-view image SEI, the process proceeds to step S207.

  If the NAL unit is neither a parameter set nor SEI, the process proceeds to step S204. In step S204, whether the NAL unit is a VCL NAL unit, that is, a decoded bit sequence in which a decoded image output order number o, a coding mode, a motion vector or a disparity vector, a coded residual signal, and the like are coded. If it is a VCL NAL unit, the process proceeds to step S208. The processing in steps S201, S202, S203, S204, and S205 corresponds to the processing operation in the separation unit 201 in the multi-viewpoint image decoding apparatus in FIG.

  When the NAL unit is a parameter set, in step S206, a parameter bit related to encoding of the entire sequence is decoded in step S206 by decoding an encoded bit sequence of parameter information related to encoding of the entire sequence and an encoded bit sequence of parameter information related to encoding of a picture. Information, parameter information related to picture coding, and the like are obtained. The processing in step S206 corresponds to the decoding operation in the parameter set decoding unit 202 in the multi-view image decoding device in FIG. Subsequently, the process returns to the separation process in step S201.

  If it is determined in step S205 that the NAL unit is a multi-view image SEI, the process proceeds to step S207, where the encoded bit sequence of the multi-view image SEI is decoded, and the encoded encoded bit sequence content is a multi-view image. And information on the number of viewpoints V are obtained. The processing in step S207 corresponds to a decoding operation in the parameter set decoding unit 202 in the multi-viewpoint image decoding apparatus in FIG. Subsequently, the process returns to the separation process in step S201.

  If it is determined in step S204 that the NAL unit is a VCL NAL unit, the following processing procedure from step S208 to step S218 is performed. First, in step S208, an encoded bit string including information such as a decoded image output order number o is decoded to obtain a decoded image output order number o. The processing in step S208 corresponds to the decoding operation of the decoded image output order number o in the encoded bit string decoding unit 204 in the multi-view image decoding device in FIG.

  Subsequently, in step S209, the viewpoint number v for specifying the viewpoint of each image from the number of viewpoints V obtained in step S207 and the decoded image output order number o obtained in step S208 (where v is 0 or more V And a number d (integer) indicating the output order of the decoded image at each viewpoint. The processing in step S209 corresponds to the decoding operation in the decoded image management information calculation unit 205 in the multi-viewpoint image decoding apparatus in FIG.

  Subsequently, in step S210, an encoded bit sequence including information such as an encoding mode, a motion / disparity vector, and an encoded residual signal (encoded prediction residual signal) is decoded and encoded. The mode, motion / disparity vector, encoded residual signal, etc. are obtained. The processing in step S208 corresponds to the encoding mode in the encoded bit string decoding unit 204 and the decoding operation of the motion / disparity vector, the encoded residual signal, and the like in the multi-view image decoding apparatus in FIG.

  Subsequently, in step S211, motion compensation prediction / disparity compensation prediction is performed from the coding mode obtained in step S210 and the motion / disparity vector to obtain a prediction block. When the pixel block is divided into small blocks or when predicted from a plurality of reference pictures, a plurality of motion compensation predictions / disparity compensation predictions are performed to obtain a plurality of prediction blocks. The processing in step S211 corresponds to the processing operation in the motion / disparity compensation prediction unit 206 in the multi-viewpoint image decoding apparatus in FIG.

  Subsequently, in step S212, when the pixel block is divided into small blocks, or when predicted from a plurality of reference pictures, a plurality of prediction blocks obtained in step S211 are synthesized, and Let it be a prediction signal. The processing in step S212 corresponds to the processing operation in the prediction signal synthesis unit 207 in the multi-viewpoint image decoding apparatus in FIG.

  Subsequently, in step S213, residual signal decoding processing such as inverse quantization and inverse orthogonal transform is performed on the encoded residual signal obtained in step S210 to generate a decoded residual signal. The processing in step S213 corresponds to the processing operation in the residual signal decoding unit 208 in the multi-viewpoint image decoding apparatus in FIG.

  Subsequently, in step S214, a decoded image signal is obtained by superimposing the decoded residual signal obtained in step S213 on the prediction signal obtained in step S212. The processing in step S214 corresponds to the processing operation in the residual signal superimposing unit 209 in the multi-viewpoint image decoding apparatus in FIG. In the next step S215, the decoded image signal prediction signal obtained in step S214 is stored in the decoding buffer. The processing in step S215 corresponds to the storing operation in the decoded image buffer 210 in the multi-viewpoint image decoding apparatus in FIG.

  Subsequently, in step S216, it is determined whether or not the decoding process has been completed for all pixel blocks in the VCL NAL unit. If completed, the process proceeds to step S217. If not completed, the process proceeds to step S210, and the processes from step S210 to step S215 are repeated until the encoding process is completed for all the pixel blocks in the VCL NAL unit.

  Subsequently, in step S217, it is determined whether or not to output a decoded image. If it is determined to output, the process proceeds to step S218, and if it is determined not to output, the process proceeds to step S219. The processing in step S217 corresponds to the management operation of the decoded image management unit 211 in the multi-viewpoint image decoding apparatus in FIG. If it is determined in step S217 to output, the process proceeds to step S218, and the decoded image is output in synchronization. The processing in step S218 corresponds to the processing operation in the decoded image output unit 212 in the multi-viewpoint image decoding apparatus in FIG.

  If it is determined in step S217 that the decoded image is not output, the process proceeds to step S219, and it is determined whether or not all the decoding processes for the encoded bit string to be decoded have been completed. If completed, this multi-viewpoint image decoding processing procedure ends. If not completed, the process returns to the first step S201, and the processes from step S201 to step S218 are repeated until all the decoding processes of the encoded bit string to be decoded are completed.

  In addition, in the multi-view image encoding apparatus of FIG. 1, a view number v that specifies the viewpoint of each image that constitutes the multi-view image signal and a number d that indicates the output order of the decoded image at each view are collectively displayed. Since the decoded image output order number o shown is encoded together with the number of viewpoints V of the multi-view image signal, the multi-view image decoding apparatus converts the encoded data encoded in this way into the decoded image output order number o. In accordance with the viewpoint number v that can be reliably decoded based on the decoded signal and that identifies the viewpoint of each viewpoint image that constitutes the multi-viewpoint image signal, and the number d that indicates the output order of the decoded image at each viewpoint, It can be appropriately output to a multi-viewpoint image display device or the like.

  In addition, when the conventional single-viewpoint image encoding / decoding method is extended to the multi-viewpoint image encoding / decoding method, the viewpoint number v for specifying the viewpoint of each viewpoint image of this method and the decoding at each viewpoint The decoded image output order number o that collectively indicates the number d indicating the output order of the images is the number that indicates the output order of the decoded images in the conventional single-viewpoint image encoding / decoding scheme (for example, picture of AVC / H.264) order count) and encoding / decoding can achieve an effect that compatibility with a conventional single-viewpoint image encoding / decoding method can be achieved with small improvements.

  The present invention is not limited to the above embodiment. For example, in the multi-view image encoding apparatus of FIG. 1, when inputting an image, the image is input to the rearrangement buffer 105 as appropriate without delay. However, if it is necessary to rearrange the pixels when storing in the rearrangement buffer 105, such as when the input image format is different from the storage format of the rearrangement buffer 105, a line buffer for temporary storage is provided. After the input image signal is temporarily written in the line buffer, the pixels are rearranged as appropriate and stored in the rearrangement buffer 105, or after the pixels are rearranged and temporarily written in the line buffer, the rearrangement buffer 105 is changed as appropriate. Can also be stored.

  Similarly, when the decoded image stored in the decoded image buffer 210 is output to the multi-view image display device or the like in the multi-view image decoding device of FIG. A line buffer for temporary storage is provided inside or outside of this, and after the decoded image signal read from the decoded image buffer 210 is temporarily written in the line buffer, the pixels are rearranged and output as appropriate, or the decoded image buffer 210 is output. The decoded image signal read out from can be rearranged in pixels and temporarily written in the line buffer, and then output as appropriate.

  In the above description of the multi-view image decoding apparatus in FIG. 17, the decoded image output unit 212 outputs the decoded image stored in the decoded image buffer 210 to the decoded image management information calculation unit 205 and the number of viewpoints V and the decoded image output. Based on the viewpoint number v that identifies the viewpoint calculated from the order number o and the number d that indicates the output order of the decoded image at each viewpoint, the decoded image of each viewpoint is controlled according to the control of the decoded image management unit 211. The signals are output in synchronization with each other. However, when the viewpoints are interleaved in units of images and serially output in one channel, they can be output based on the decoded image output order number o. In this case, the viewpoint number v or viewpoint ID for specifying the viewpoint of each image is associated with the decoded image signal of each viewpoint in order from the smallest decoded image output order number o of each decoded image, and is output together with these parameters. By doing so, the decoded image signal of each viewpoint can be interleaved in units of images and output in synchronization with each other. The output order in units of images described here is equivalent to the output order in units of images described with reference to FIG. Thus, even when a decoded image is output according to the decoded image output order number o of each decoded image, each image calculated by the decoded image management information calculation unit 205 from the viewpoint number V and the decoded image output order number o. The decoded image signal of each viewpoint can be appropriately output by associating with the viewpoint number v or the viewpoint ID for specifying the viewpoint, and simultaneously outputting them together with these parameters, and the multi-view image display of the output destination of the decoded image The correspondence between the viewpoint and the image can be grasped by an apparatus or the like.

  In the above description, multi-viewpoint images used for encoding and decoding can be encoded and decoded from multi-viewpoint images actually captured from different viewpoints. It is also possible to encode and decode a viewpoint image that has been converted or generated, such as by interpolating the viewpoint position from surrounding viewpoints, and is included in the present invention. In addition, multi-view images such as computer graphics can be encoded and decoded, and is included in the present invention.

  For example, a multi-viewpoint image signal including image signals of four viewpoints A, B, C, and D is (1) an image signal obtained by actually photographing all four viewpoint image signals at each viewpoint. In some cases, (2) when all four viewpoint image signals are virtually taken at each viewpoint, (3) A and B viewpoint image signals are actually captured at each viewpoint. The image signal obtained by actually shooting and the image signal obtained by actually shooting, such as the image signal obtained by virtually capturing the image signal of the C and D viewpoints and the image signals of the C and D viewpoints. There are three cases where the generated image signal and the generated image signal are mixed.

  In addition, the position of each viewpoint of the multi-viewpoint image used in the present invention may be any arrangement. This will be described with reference to FIGS. The numbers shown in FIGS. 21 to 24 are viewpoint numbers v (v = 0, 1, 2,...) Indicating viewpoint positions. FIG. 21 shows an example in which the viewpoints are arranged in the horizontal direction. The images were taken with the cameras arranged horizontally. FIG. 22 shows an example in which the viewpoints are arranged in the vertical direction. The picture was taken with the cameras arranged vertically. 23 and 24 show examples in which the viewpoints are arranged in a two-dimensional horizontal / vertical direction. The images were taken with the cameras arranged in two horizontal / vertical directions. The value of the viewpoint number v that identifies the viewpoint has a one-to-one correspondence with each viewpoint and must be assigned an integer that is greater than or equal to 0 and less than the number V of viewpoints. Any order is acceptable.

  The above multi-view image encoding and decoding processes can be realized as a transmission, storage, and reception device using hardware, as well as a ROM (Read Only Memory) and a flash memory. It can also be realized by firmware stored in the computer or software such as a computer. When realized as software such as a computer, a RAM (Random Access Memory) generally used on the computer can be used as a rearrangement buffer and a decoded image buffer. In addition, when the encoded coded bit string is transmitted through the network, it can be transmitted through a network interface or the like mounted on the computer. The firmware program and software program can be recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as a data broadcast of terrestrial or satellite digital broadcasting Is also possible.

It is a block diagram of an example of the multi-view image encoding apparatus which produces | generates the encoding data of the multi-view image received by this invention. 3 is a flowchart for explaining multi-view image encoding processing in FIG. 1. It is a figure explaining an example of the encoding syntax structure of 1st multiview image SEI. It is a figure explaining an example at the time of assigning a value to the viewpoint number v which specifies the viewpoint of each image, and the number d which shows the output order of the decoded image in each viewpoint. It is a figure explaining an example at the time of assigning a value to decoded image output order number o of each 1st picture. It is a figure explaining an example at the time of assigning a value to decoded image output order number o of each 2nd picture. It is a figure explaining an example at the time of interleaving the signal of each viewpoint per pixel. It is a figure explaining an example at the time of interleaving the signal of each viewpoint by the unit which put together the several pixel. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in pixel block units, such as 16x16 and 16x8 pixels. It is a figure explaining an example at the time of interleaving the signal of each viewpoint by the line unit of a horizontal direction. It is a figure explaining an example at the time of interleaving in the form which put together the signal of each viewpoint in one picture. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in the slice unit which put together the some line. It is a figure explaining an example at the time of interleaving the signal of each viewpoint for every image. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in the unit which put together the some image. It is a figure explaining an example of the encoding relationship of each image, and the reference relationship of motion compensation / parallax compensation. 3 is a flowchart for explaining multiplexing processing of FIG. 2. It is a block diagram of one Embodiment of the multiview image decoding apparatus which is the principal part of the multiview image receiving apparatus of this invention. 18 is a flowchart for explaining the multi-view image decoding process of FIG. 17. It is a figure explaining an example of the encoding syntax structure of the 2nd multi viewpoint image SEI. FIG. 19 is a flowchart for explaining separation processing in FIG. 18. FIG. It is a figure explaining an example at the time of arrange | positioning a viewpoint to a horizontal direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint to a perpendicular direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint in a horizontal / vertical two-dimensional direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint in a horizontal / vertical two-dimensional direction. It is a block diagram of an example of the conventional stereoscopic vision multi-view image encoding apparatus. It is a block diagram of an example of the conventional stereoscopic vision image decoding apparatus.

Explanation of symbols

Reference Signs List 101 encoding management unit 102 parameter set encoding unit 103 multi-view image SEI encoding unit 104 decoded image output order number calculation unit 105 rearrangement buffer 106 motion / disparity compensation prediction unit 107 encoding mode determination unit 108 residual signal calculation unit 109 Residual Signal Encoding Unit 110 Residual Signal Decoding Unit 111 Residual Signal Superimposing Unit 112 Decoded Image Buffer 113 Encoded Bit String Generation Unit 114 Multiplexing Unit 201 Separating Unit 202 Parameter Set Decoding Unit 203 Multi-View Image SEI Decoding Unit 204 Code Bit stream decoding unit 205 Decoded image management information calculation unit 206 Motion / disparity compensation prediction unit 207 Prediction signal synthesis unit 208 Residual signal decoding unit 209 Residual signal superimposition unit 210 Decoded image buffer 211 Decoded image management unit 212 Decoded image output unit

Claims (6)

It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image receiving method for receiving encoded data obtained by encoding a multi-view image signal which is an image signal generated as a virtual image taken from one viewpoint, and decoding the encoded data,
Supplementary additional information indicating that the multi-view image signal is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint, and the decoded image at each viewpoint A first step of receiving encoded data in which a decoded image output order number o (where o is an integer equal to or greater than 0) and the multi-view image signal are encoded, each indicating a number d indicating an output order; ,
First encoded data in which the supplementary additional information and the number of viewpoints V of the multi-view image signal are encoded from the encoded data received in the first step, and the decoded image output order A second step of separating the second encoded data in which the number o and the multi-view image signal are respectively encoded;
A third step of decoding the first encoded data to generate the supplementary additional information including the number of viewpoints V of the multi-view image signal;
A fourth step of decoding the second encoded data to generate the decoded image output order number o;
A number indicating the output order of the decoded image at each viewpoint obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints (integer of 0 or more). a fifth step of calculating as a number v specifying each of the viewpoints, and calculating the remainder of the division (an integer greater than or equal to 0 and less than V);
A sixth step of decoding the second encoded data to generate a decoded multi-viewpoint image signal;
A seventh step of storing the decoded multi-viewpoint image signal in an image buffer;
According to the number d indicating the output order of the decoded images at the respective viewpoints calculated at the fifth step and the number v specifying each of the viewpoints, the decoded multi-viewpoint images are extracted from the image buffer. An eighth step of extracting a signal and outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other;
A multi-viewpoint image receiving method. It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image receiving method for receiving encoded data obtained by encoding a multi-view image signal which is an image signal generated as a virtual image taken from one viewpoint, and decoding the encoded data,
For supplementary additional information indicating that the multi-view image is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the management of the decoded image A decoded image output order number o (o is a batch indicating a correspondence relationship with viewpoint IDs that specify viewpoints, a number v that identifies each viewpoint, and a number d that indicates the output order of decoded images at each viewpoint. A first step of receiving encoded data in which each of the multi-view image signals is encoded,
Correspondence between the supplementary information, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the viewpoint ID from the encoded data received in the first step A second that separates the first encoded data in which the relationship is encoded, and the second encoded data in which the decoded image output order number o and the multi-view image signal are encoded, respectively. And the steps
The supplementary addition including the correspondence between the number V of viewpoints of the multi-view image signal and the number v for identifying each viewpoint of the multi-view image and the viewpoint ID by decoding the first encoded data A third step of generating information;
A fourth step of decoding the second encoded data to generate the decoded image output order number o;
A quotient (integer) obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints is calculated as a number d indicating the output order of the decoded images at each viewpoint. And a fifth step of calculating the remainder of the division (an integer greater than or equal to 0 and less than V) as a number v that identifies each of the viewpoints;
A sixth step of decoding the second encoded data to generate a decoded multi-viewpoint image signal;
A seventh step of storing the decoded multi-viewpoint image signal in an image buffer;
According to the number d indicating the output order of the decoded images at the respective viewpoints calculated in the fifth step and the viewpoint ID, a decoded multi-view image signal is extracted from the image buffer, and the decoded multi-viewpoints are obtained. An eighth step of outputting the decoded image signals of the respective viewpoints constituting the image signal in synchronization with each other;
A multi-viewpoint image receiving method. It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image receiving device that receives encoded data obtained by encoding a multi-view image signal that is an image signal generated as a virtual image taken from one viewpoint, and decodes the encoded data.
Supplementary additional information indicating that the multi-view image is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint, and the output of the decoded image at each viewpoint Receiving means for receiving encoded data in which a decoded image output order number o (o is an integer equal to or greater than 0) and the multi-view image signal are respectively encoded, which collectively indicate a number d indicating the order;
The supplementary additional information indicating that the multi-view image is encoded and the number of viewpoints V of the multi-view image signal are encoded from the encoded data received by the receiving unit. Separating means for separating encoded data and second encoded data in which the decoded image output order number o and the multi-view image signal are encoded;
First decoding means for decoding the separated first encoded data and generating the supplementary additional information including the number of viewpoints V of the multi-view image signal;
Second decoding means for decoding the separated second encoded data and generating the decoded image output order number o;
A quotient (integer) obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints is calculated as a number d indicating the output order of the decoded images at each viewpoint. And calculating means for calculating the remainder of the division (an integer of 0 or more and less than V) as a number v for identifying each of the viewpoints;
Third decoding means for decoding the second encoded data and generating a decoded multi-viewpoint image signal;
Storage means for storing the decoded multi-viewpoint image signal in an image buffer;
In accordance with a number d indicating the output order of the decoded images at the respective viewpoints supplied from the calculation means and a number v specifying each of the viewpoints, a decoded multi-viewpoint image signal is extracted from the image buffer, Output means for outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other;
A multi-viewpoint image data receiving apparatus comprising: It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image receiving device that receives encoded data obtained by encoding a multi-view image signal that is an image signal generated as a virtual image taken from one viewpoint, and decodes the encoded data.
For supplementary additional information indicating that the multi-view image is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the management of the decoded image A decoded image output order number o (o is a batch indicating a correspondence relationship with viewpoint IDs that specify viewpoints, a number v that identifies each viewpoint, and a number d that indicates the output order of decoded images at each viewpoint. Receiving means for receiving encoded data in which each of the multi-view image signals is encoded,
From the received encoded data, the supplementary additional information, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the correspondence relationship between the viewpoint IDs are encoded. Separating means for separating the first encoded data and the second encoded data in which the decoded image output order number o and the multi-view image signal are encoded respectively;
The separated first encoded data is decoded, and the number V of viewpoints of the multi-view image signal, the number v for specifying each viewpoint of the multi-view image, and the viewpoint for management of the decoded image are specified. First decoding means for generating the supplementary additional information including a correspondence relationship with the viewpoint ID to be performed;
Second decoding means for decoding the separated second encoded data and generating the decoded image output order number o;
A quotient (integer) obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints is calculated as a number d indicating the output order of the decoded images at each viewpoint. And calculating means for calculating the remainder of the division (an integer of 0 or more and less than V) as a number v for identifying each of the viewpoints;
Third decoding means for decoding the second encoded data and generating a decoded multi-viewpoint image signal;
Storage means for storing the decoded multi-viewpoint image signal in an image buffer;
According to the number d indicating the output order of the decoded image at each viewpoint supplied from the calculation means and the viewpoint ID, the decoded multi-view image signal is extracted from the image buffer, and the decoded multi-view image signal is Output means for outputting the decoded image signals of the respective viewpoints that are configured to be synchronized with each other;
A multi-viewpoint image data receiving apparatus comprising: It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or Multi-view image reception for receiving encoded data obtained by encoding a multi-view image signal, which is an image signal virtually generated from one viewpoint, using a computer and decoding the received encoded data A program for
In the computer,
Supplementary additional information indicating that the multi-view image signal is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint, and the decoded image at each viewpoint A first step of receiving encoded data in which a decoded image output order number o (where o is an integer equal to or greater than 0) and the multi-view image signal are encoded, each indicating a number d indicating an output order; ,
First encoded data in which the supplementary additional information and the number of viewpoints V of the multi-view image signal are encoded from the encoded data received in the first step, and the decoded image output order A second step of separating the second encoded data in which the number o and the multi-view image signal are respectively encoded;
A third step of decoding the first encoded data to generate the supplementary additional information including the number of viewpoints V of the multi-view image signal;
A fourth step of decoding the second encoded data to generate the decoded image output order number o;
A number indicating the output order of the decoded image at each viewpoint obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints (integer of 0 or more). a fifth step of calculating as a number v specifying each of the viewpoints, and calculating the remainder of the division (an integer greater than or equal to 0 and less than V);
A sixth step of decoding the second encoded data to generate a decoded multi-viewpoint image signal;
A seventh step of storing the decoded multi-viewpoint image signal in an image buffer;
According to the number d indicating the output order of the decoded images at the respective viewpoints calculated at the fifth step and the number v specifying each of the viewpoints, the decoded multi-viewpoint images are extracted from the image buffer. An eighth step of extracting a signal and outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other;
A program for receiving multi-viewpoint images, characterized in that It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or Multi-view image reception for receiving encoded data obtained by encoding a multi-view image signal, which is an image signal virtually generated from one viewpoint, using a computer and decoding the received encoded data A program for
In the computer,
For supplementary additional information indicating that the multi-view image is encoded, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the management of the decoded image A decoded image output order number o (o is a batch indicating a correspondence relationship with viewpoint IDs that specify viewpoints, a number v that identifies each viewpoint, and a number d that indicates the output order of decoded images at each viewpoint. A first step of receiving encoded data in which each of the multi-view image signals is encoded,
Correspondence between the supplementary information, the number V of viewpoints of the multi-view image signal, the number v for identifying each viewpoint of the multi-view image, and the viewpoint ID from the encoded data received in the first step A second that separates the first encoded data in which the relationship is encoded, and the second encoded data in which the decoded image output order number o and the multi-view image signal are encoded, respectively. And the steps
The supplementary addition including the correspondence between the number V of viewpoints of the multi-view image signal and the number v for identifying each viewpoint of the multi-view image and the viewpoint ID by decoding the first encoded data A third step of generating information;
A fourth step of decoding the second encoded data to generate the decoded image output order number o;
A quotient (integer) obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints is calculated as a number d indicating the output order of the decoded images at each viewpoint. And a fifth step of calculating the remainder of the division (an integer greater than or equal to 0 and less than V) as a number v that identifies each of the viewpoints;
A sixth step of decoding the second encoded data to generate a decoded multi-viewpoint image signal;
A seventh step of storing the decoded multi-viewpoint image signal in an image buffer;
According to the number d indicating the output order of the decoded images at the respective viewpoints calculated in the fifth step and the viewpoint ID, a decoded multi-view image signal is extracted from the image buffer, and the decoded multi-viewpoints are obtained. An eighth step of outputting the decoded image signals of the respective viewpoints constituting the image signal in synchronization with each other;
A program for receiving multi-viewpoint images, characterized in that

Images