JP2008034893A - Multi-viewpoint image decoder - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem wherein a buffer for rearrangement for each viewpoint is required and delay may be generated since a decoded image signal is outputted for each frame or for each field and then is delayed for each frame or for each field successively for synchronization in a conventional stereoscopic image decoder. <P>SOLUTION: A decoding image management section 208 controls a decoding buffer 207 based on the number V of viewpoints supplied from a decoding image management information calculation section 202, a viewpoint number v for specifying the viewpoint of each viewpoint image, and a number d for indicating the output order of the decoding image at each viewpoint. A decoding image output section 209 outputs a decoding image stored into the decoding image buffer 207 to a multi-viewpoint image decoder, or the like as a viewpoint image M(v) by synchronizing each viewpoint mutually based on the viewpoint number v for specifying the viewpoint of each viewpoint image calculated from the number V of viewpoints and a decoding image output order number o at the decoding image management information calculation section 202 and the number d for indicating the output order of the decoding image at each viewpoint. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a multi-view image decoding apparatus, and more particularly, to a multi-view image decoding apparatus that decodes encoded multi-view image data shot and encoded from different viewpoints.

<Video coding system>
Currently, moving images on the time axis are handled as digital signal information. At that time, motion compensated prediction is used using redundancy in the time direction for the purpose of broadcasting, transmitting or storing information with high efficiency. In addition, apparatuses and systems that comply with an encoding scheme such as MPEG (Moving Picture Experts Group) that encodes and compresses using orthogonal transformation such as discrete cosine transformation using redundancy in the spatial direction have become widespread.

  The MPEG-2 video (ISO / IEC 13818-2) encoding system established in 1995 is defined as a general-purpose moving image compression encoding system, and supports interlaced scanned images in addition to progressive scanned images. Supports not only SDTV (standard definition images) but also HDTV (high definition images), and storage of DVDs (Digital Versatile Disks), which are optical discs, and magnetic tapes using D-VHS (registered trademark) digital VTRs. It is widely used as an application for media and digital broadcasting.

  In addition, MPEG-4 visual (ISO / IEC 14496-2) encoding method was standardized, aiming at higher encoding efficiency in applications such as network transmission and portable terminals, and was established as an international standard in 1998. It was done.

Furthermore, in 2003, joint work of the International Technical Organization (ISO) and the International Electrotechnical Commission (IEC) Joint Technical Committee (ISO / IEC) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) -4 AVC / H.264 encoding method (ISO / IEC 1449-10, ITU-T H.264 standard number. This is hereinafter referred to as AVC / H.264 encoding method. Called the international standard.
This AVC / H.264 encoding method achieves higher encoding efficiency than conventional encoding methods such as MPEG-2 video and MPEG-4 visual.

  In a P picture of an encoding system such as MPEG-2 video or MPEG-4 visual, motion compensation prediction is performed only from the immediately preceding I picture or P picture in the display order. On the other hand, in the AVC / H.264 encoding method, a plurality of pictures can be used as reference pictures, and motion compensation can be performed by selecting an optimum one for each block. Further, in addition to the preceding picture in the display order, a subsequent picture can be referred to in the already encoded display order. In addition, a B picture of an encoding method such as MPEG-2 video or MPEG-4 visual refers to one reference picture in the display order, one reference picture in the rear, or two reference pictures at the same time. The average value of the two pictures is a predicted picture, and the difference data between the target picture and the predicted picture is encoded.

  On the other hand, in the AVC / H.264 encoding method, any reference picture can be referred to for prediction regardless of the front or rear, regardless of the restriction of one front and one rear in the display order. Furthermore, it is possible to refer to the B picture as a reference picture.

  In addition, AVC / H. In the H.264 encoding method, when a decoded image is output, the reference picture and the non-reference picture are changed in order from the encoding order to the output order of the decoded picture (desired order when displayed on the display device of the output destination). Both have to be stored in the memory, but a mechanism has been introduced in which the reference picture and the non-reference picture are unifiedly managed by a single memory called a decoded picture buffer. The coded sequence (bit stream) is encoded with information indicating an output order called a picture order count for each image, and is numbered in the output order of the decoded images. It has been. The images that have been decoded and stored in the decoded image buffer are sequentially output from the images with the smallest picture order count value. Also, the picture order count is reset with an IDR picture (a picture that means that subsequent pictures can be normally decoded without using information of pictures preceding the picture in the coding order).

<Multi-view image coding method>
On the other hand, in a twin-lens stereoscopic television, a left-eye image and a right-eye image captured from two different directions by two cameras are generated and displayed on the same screen to show a stereoscopic image. I have to. In this case, the left eye image and the right eye image are separately transmitted or recorded as independent images. However, this requires about twice as much information as a single two-dimensional image.

  Therefore, conventionally, a method has been proposed in which one of the left and right images is set as a main image and the other image (sub-image) information is information-compressed by a general compression encoding method to reduce the amount of information (for example, Patent Document 1). In the stereoscopic television image transmission method described in Patent Document 1, a relative position with high correlation in the other image is obtained for each small region, and the position shift amount (parallax vector) and a difference signal (prediction residual signal) are obtained. ). The difference signal is also transmitted and recorded because the image close to the sub-image can be restored using the main image and the amount of disparity and position shift, which is parallax information, but there is no main image such as the shadow of the object. This is because the sub-image information cannot be restored.

  In 1996, a stereo image encoding method called a multi-view profile was added to the MPEG-2 video (ISO / IEC 14496-2) encoding method, which is an international standard for single-view image encoding (ISO / IEC). IEC 14496-2 / AMD3). The MPEG-2 video multi-view profile is a two-layer encoding method that encodes the image for the left eye with the base layer and the image for the right eye with the enhancement layer, and motion compensated prediction using redundancy in the time direction In addition to discrete cosine transformation using redundancy in the spatial direction, encoding compression is performed using disparity compensation prediction using redundancy between viewpoints.

  Furthermore, a method using one system encoding path and decoding path when encoding / decoding a multi-viewpoint image has been proposed (for example, see Patent Document 2). In this method, on the encoding side (transmission side), image signals of a plurality of channels obtained by photographing from a plurality of directions as shown in FIG. Delay one field at a time. The control unit 304 outputs a switch signal having a frame period or a field period, and controls the multiplexer 302 so as to sequentially extract the signals of a plurality of channels from the sequential unit 301.

  The multiplexer 302 sequentially inserts one frame or one field at a time to form one image signal. The encoding unit 303 encodes the image signal output from the multiplexer 302 and outputs an encoded bit string to the transmission path.

  On the decoding side (receiving side), the encoded bit string is decoded by the decoding means 305 as shown in FIG. The control unit 308 controls the demultiplexer 306 by outputting a switch signal having a frame period or a field period from the decoded image signal. As a result, each frame or field of the decoded image signal is sequentially extracted from the demultiplexer 306 and supplied to the synchronizer 307, where the frames or fields are sequentially delayed and synchronized one frame or one field at a time. It outputs as a similar multi-channel image signal.

JP-A 61-144191 Japanese Patent Laid-Open No. 10-243419

  However, in the conventional stereoscopic image encoding / decoding method and apparatus, the image signals of a plurality of channels constituting the multi-view image signal at the time of encoding are sequentially arranged for each frame or one field. Since it is input to the encoding unit 303 as one image signal and further encoded and rearranged in the encoding order by the encoding unit 303, a buffer for sequentially rearranging every frame or one field is necessary. There may be a delay.

  Also, at the time of decoding, the image signals decoded by the decoding means 305 are sequentially arranged and output for each frame or field, and then sequentially delayed by one frame or field, so that the viewpoint is synchronized. A buffer for rearrangement is required every time, and a delay may occur at that time.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a multi-viewpoint image decoding apparatus that reduces the buffer size and shortens the delay time.

  In order to achieve the above object, the first invention is a multi-view image signal including image signals of respective viewpoints respectively obtained from a plurality of set viewpoints, and the image signal of one viewpoint is actually transmitted from one viewpoint. A multi-viewpoint image decoding apparatus that decodes encoded data obtained by encoding a multi-viewpoint image signal that is an image signal obtained by shooting in the image or an image signal generated as a virtual shot from one viewpoint. Decoding means for decoding the encoded data to generate a decoded multi-view image signal; storage means for storing the decoded multi-view image signal in the image buffer as needed; taking out the decoded multi-view image signal from the image buffer; And output means for outputting the decoded image signals as independent channels for each decoded image signal of each viewpoint.

  In the present invention, when the input format of the display device or the like is a parallel input format of a plurality of channels, the decoded multi-viewpoint image signal stored in the image buffer is matched with the input format of the display device or the like for each viewpoint. Can be output in independent channels, so that it is not necessary to synchronize after sequentially arranging and outputting every frame or one field after decoding as in the conventional stereoscopic image decoding method and apparatus, A decoded image signal for each viewpoint can be obtained.

  In order to achieve the above object, the second invention is a multi-viewpoint image signal including image signals of respective viewpoints respectively obtained from a plurality of set viewpoints, and an image signal of one viewpoint is Multi-viewpoint image decoding apparatus that decodes encoded data obtained by encoding a multi-viewpoint image signal that is an image signal that is actually captured from a viewpoint or an image signal that is virtually captured from one viewpoint A decoding unit that decodes encoded data to generate a decoded multi-view image signal; a storage unit that stores the decoded multi-view image signal in an image buffer as needed; and a decoded multi-view image signal is extracted from the image buffer Output means for synchronizing the decoded image signals of the respective viewpoints with each other, interleaving the decoded image signals of the respective viewpoints, and outputting them serially in one channel.

  In this invention, when the input format of the display device or the like is a serial input format of one channel, the decoded multi-viewpoint image signal stored in the image buffer is decoded for each viewpoint according to the input format of the display device or the like. Since image signals can be synchronized with each other and interleaved and output serially in one channel, they are sequentially arranged and output for each frame or field after decoding as in the conventional stereoscopic image decoding method and apparatus. Then, there is no need to synchronize, and a decoded image signal for each viewpoint can be obtained.

  Furthermore, in order to achieve the above object, the third invention is a multi-view image signal including the image signals of the respective viewpoints respectively obtained from a plurality of set viewpoints. Multi-viewpoint image decoding apparatus that decodes encoded data obtained by encoding a multi-viewpoint image signal that is an image signal that is actually captured from a viewpoint or an image signal that is virtually captured from one viewpoint The multi-view image signal, the number V of viewpoints of the multi-view image signal, the number v that identifies each viewpoint, and the number d that indicates the output order of the decoded image at each viewpoint are collectively shown. Decoded encoded image output order numbers o (integers) are encoded respectively, and decoded multi-view image signals, the number V of viewpoints of the multi-view image signals, and decoded image output order numbers o Each of the decoding means for generating The quotient (integer) obtained by dividing the decoded decoded image output order number o by the number V of decoded viewpoints by integer arithmetic is calculated as a number d indicating the output order of the decoded images at the respective viewpoints. , A calculation means for calculating the remainder of division (an integer greater than or equal to 0 and less than V) as a number v for identifying each viewpoint, a storage means for storing the decoded multi-viewpoint image signal in the image buffer, and a calculation means. The decoded multi-viewpoint image signal is extracted from the image buffer in accordance with the number d indicating the output order of the decoded image at each viewpoint and the number v specifying each viewpoint, and each viewpoint constituting the decoded multi-viewpoint image signal is extracted. Output means for outputting the decoded image signals in synchronization with each other.

  In the present invention, the number V of viewpoints of the multi-viewpoint image signal, the number v that identifies each viewpoint, and the decoded image output order number o that collectively indicates the number d that indicates the output order of the decoded image at each viewpoint. The number v for identifying the viewpoint of each image signal of each viewpoint constituting the multi-view image signal based on the decoded image output order number o and the number V of decoded viewpoints, and the decoded image at each viewpoint And the number d indicating the output order of the image can be appropriately output to a multi-view image display device or the like, and the conventional single-view image encoding / decoding method can be applied to the multi-view image code of the present invention. When expanding to a decoding / decoding method, the decoded image output order number o of the present invention is treated as a number indicating the output order of decoded images of the conventional encoding method (for example, picture order count in the AVC / H.264 method) Can .

  According to the present invention, unlike the conventional stereoscopic image decoding method and apparatus, there is no need to synchronize after sequentially arranging and outputting every frame or one field after decoding, and an image buffer for synchronization The delay time can be shortened.

  Further, according to the present invention, the decoded image that collectively indicates the viewpoint number v that specifies the viewpoint of the image signal of each viewpoint constituting the multi-viewpoint image signal and the number d that indicates the output order of the decoded image at each viewpoint. Since the encoded data obtained by encoding the output order number o together with the number V of viewpoints of the multi-view image signal is decoded, the viewpoint number v for specifying the viewpoint of the image signal of each viewpoint constituting the multi-view image signal; According to the number d indicating the output order of the decoded image at each viewpoint, it can be appropriately output to a multi-view image display device or the like.

  Thus, according to the present invention, when the conventional single-view image encoding / decoding method is expanded to the multi-view image encoding / decoding method, the decoded image output order number o of the present invention is changed to the conventional code. By processing and decoding as a number indicating the output order of the decoded image of the encoding method (for example, picture order count in the AVC / H.264 method), compatibility with the conventional encoding / decoding method can be achieved with a small improvement. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, a multi-view image encoding device that generates encoded data to be decoded by the multi-view image decoding device of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an example of the multi-view image encoding device described above, and FIG. 2 is a flowchart for explaining a processing procedure of the multi-view image encoding device of FIG.

  As shown in FIG. 1, the multi-view image encoding apparatus includes an encoding management unit 101, a decoded image output order number calculation unit 102, a rearrangement buffer 103, a motion / disparity compensation prediction unit 104, an encoding mode determination unit 105, a remaining A difference signal calculation unit 106, a residual signal encoding unit 107, a residual signal decoding unit 108, a residual signal superimposing unit 109, a decoded image buffer 110, and an encoded bit string generation unit 111 are provided.

  Next, the operation of the multi-view image encoding apparatus shown in FIG. 1 will be described with reference to the flowchart of FIG. First, the encoding management unit 101 in FIG. 1 is supplied with camera parameter information such as the number of viewpoints of a multi-view image supplied to the multi-view image encoding apparatus, and the position and orientation of the camera at each viewpoint at the time of shooting. . These pieces of information are necessary for displaying the multi-viewpoint image on the receiving side, and the viewpoint number V of the multi-viewpoint image and the camera parameter information are encoded bit sequence via the decoded image output order number calculation unit 102. The data is supplied to the generation unit 111 and is generated as an encoded bit string (step S101 in FIG. 2).

  Further, the encoding management unit 101 is supplied with information for specifying each viewpoint, information such as a time stamp, etc. for each of the images constituting the multi-viewpoint image. Based on these information and image input order, etc., which viewpoint each image belongs to, the output order of the decoded image at each viewpoint (the encoded bit string obtained by encoding with this multi-viewpoint image encoding device) (Decoding image output order at each viewpoint obtained by decoding on the decoding side), synchronization between each viewpoint, and the like are managed. For each image, a viewpoint number v that identifies the viewpoint and a number d that indicates the output order of the decoded image at each viewpoint are assigned, and are associated with each image.

Here, how to assign the value of the viewpoint number v that identifies each viewpoint will be described. As the viewpoint number v, 0 or a positive integer less than V indicating the number of viewpoints is assigned to each viewpoint.
The same value is assigned to images of the same viewpoint, and different values are assigned to images of different viewpoints. As described above, by assigning a value to the viewpoint number v of each image, it is possible to specify which viewpoint each image belongs to by the viewpoint number v.

  Next, a description will be given of how to assign the value of the number d indicating the output order of the decoded image at each viewpoint. An integer is assigned to each number d, and the value is increased in accordance with the output order of decoded images. However, when the output times of the decoded images between the viewpoints are the same, the values of the numbers d are the same. As described above, by assigning a value to the number d indicating the output order of each decoded image, the decoding side outputs from the image having a smaller number d according to the value of the number d, and outputs in the desired output order. In addition, when the number d is the same between the viewpoints, it can be determined that the output times are the same, so that the viewpoints can be output in synchronization. In addition, the coding management unit 101 manages the coding order of each image and manages reference images used for motion compensation prediction / parallax compensation prediction.

  FIG. 3 is a view number v that identifies the viewpoint and a number d that indicates the output order of the decoded image at each viewpoint for each of the images constituting the multi-view image encoded by the multi-view image encoding device of FIG. It is a figure which shows an example at the time of assigning a value to. In the figure, the vertical axis represents the viewpoint direction, and the horizontal axis represents the time direction. M (v) (v = 0, 1, 2,..., V−1) indicates viewpoint images constituting the multi-viewpoint image, and v is a viewpoint number for specifying each viewpoint. . Further, m (v, d) (v = 0, 1, 2,..., V−1; d = 0, 1, 2,...) Represents an image constituting the viewpoint image M (v). In the figure, v is a viewpoint number that identifies each viewpoint, and d is a number that indicates the output order of the decoded image at each viewpoint. For example, when comparing two images m (v, d), if the values of the numbers v are the same, they are images of the same viewpoint, and if the values of the numbers v are different, both Are images of different viewpoints. When the values of the numbers d are the same, the images are the same time. When the values of the numbers d are different, the images are at different times, and the smaller value is the image at the earlier time.

  In addition to encoding the number of viewpoints V of the multi-viewpoint image signal, it is also possible to individually encode the viewpoint number v for specifying the viewpoint and the number d indicating the output order of the decoded image at each viewpoint. In the present invention, encoding is performed as a decoded image output order number o indicating both of them collectively. When the conventional single-viewpoint image encoding / decoding system is extended to the multi-viewpoint image encoding / decoding system of the present embodiment, the viewpoint number v for specifying the viewpoint of this system and the decoded image at each viewpoint A decoded image output order number o that collectively indicates a number d indicating the output order of the image is treated as a number indicating the output order of the decoded image of the conventional single-viewpoint image encoding / decoding method, and is encoded / decoded. With a small improvement, compatibility with the conventional single-viewpoint image encoding / decoding method can be achieved. Specifically, when the AVC / H.264 system is expanded to the multi-viewpoint encoding system, the decoded image output order number o of this system is a number indicating the output order of the decoded image of the AVC / H.264 system. Treat as a picture order count.

  The decoded image output order number calculation unit 102 in FIG. 1 outputs the decoded image from the viewpoint number V of the multi-view image to be encoded, the viewpoint number v that identifies each viewpoint, and the output order d of the decoded image at each viewpoint. A sequence number o is calculated (step S103 in FIG. 2). The decoded image output order number o is calculated by the following equation.

o = d · V + v (1)
4 calculates a decoded image output order number o for each of the images constituting a multi-view image of five viewpoints (V = 5) to be encoded by the multi-view image encoding device of FIG. It is a figure which shows an example at the time of assigning a value. The encoded bit string generation unit 111 encodes the decoded image output order number o calculated by the decoded image output order number calculation unit 102 into a bit string (step S104 in FIG. 2).

  Further, in the multi-view image decoding apparatus described later on the decoding side, from the viewpoint number V of the multi-view image obtained by decoding the bit string encoded by the multi-view image encoding apparatus in FIG. 1 and the decoded image output order number o. A viewpoint number v (where v is an integer greater than or equal to 0 and less than V) and a number d (integer) indicating the output order of decoded images at each viewpoint are calculated. Specifically, the number d is a quotient obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. The number v is a remainder value obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. Alternatively, the viewpoint number v may be calculated by the following equation after calculating the number d.

v = od−V (2)
The rearrangement buffer 103 stores the supplied multi-viewpoint images. Here, the viewpoint images M (v) (v = 0, 1, 2,..., V−1) constituting the multi-viewpoint image are input in parallel through independent channels for each viewpoint, There is a method in which a viewpoint image signal is serially input through one channel as an interleaved signal. As a method of interleaving the image signals of each viewpoint, a method of interleaving the image signals of each viewpoint in units of pixels, a method of interleaving in units of a plurality of pixels, a method of interleaving in units of horizontal lines, a unit of images There are a method of interleaving, a method of interleaving in units of a plurality of images, and the like.

  An example of the interleave structure of the input viewpoint image M (v) will be described with reference to FIGS. FIG. 5 shows an example in which the signals at each viewpoint are interleaved in units of pixels. In the figure, p (v, i) represents a pixel of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and i is a pixel index.

FIG. 6 shows an example of interleaving the signals of each viewpoint in units of a plurality of pixels.
In the figure, py (v, iy) represents a pixel of the luminance signal of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and iy is an index of the pixel of the luminance signal. pu (v, iu) represents a pixel of the color difference signal (U), and iu is an index of the pixel of the color difference signal (U). pv (v, iv) represents a pixel of the color difference signal (V), and iv is an index of the pixel of the color difference signal (V).

  FIG. 7 shows an example in which the signals of each viewpoint are interleaved in units of pixel blocks of 16 pixels in the horizontal direction and 16 pixels in the vertical direction, or in units of pixel blocks of 16 pixels in the horizontal direction and 8 pixels in the vertical direction. In the figure, b (v, ib) represents a pixel of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and ib is a pixel block index. It can be said that this is a type of interleaving in a unit of a plurality of pixels by scanning a pixel block along a horizontal line.

  FIG. 8 shows an example of interleaving the signals of each viewpoint in units of horizontal lines. In the figure, l (v, j) represents a line of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and j is an index of the line. Since a line is composed of a plurality of pixels, it can be said that the line is a kind of interleaved unit of a plurality of pixels.

  Further, FIG. 9 can be said to be a type of interleaving in units of horizontal lines by scanning the images combined into one image in a horizontal line even when the signals of each viewpoint are interleaved in a format combined into one image. FIG. 9A shows one image in which the signals of the respective viewpoints are collected, and FIG. 9B shows an image interleaved in units of horizontal lines. In the same figure, v indicates a viewpoint number for specifying a viewpoint, and d indicates a number indicating an imaging order of images at each viewpoint (a number indicating an output order of decoded images at each viewpoint). Further, j in l (v, j) indicates a line index.

  FIG. 10 shows an example in which the signals of each viewpoint are interleaved in units of slices in which a plurality of lines are combined. In the figure, s (v, k) represents a pixel of a luminance signal of each viewpoint image, v is a viewpoint number for specifying the viewpoint, and k is an index of a slice in which a plurality of lines are collected. Since a slice is composed of a plurality of pixels, it can be said that it is a kind of interleaving in units of a plurality of pixels and also a kind of interleaving in units of pixel blocks.

  FIG. 11 shows an example in which the signals of each viewpoint are interleaved in units of images. In the figure, m (v, d) represents an image constituting each viewpoint image, v is a viewpoint number for specifying the viewpoint, and d is a number indicating an image capturing order at each viewpoint (at each viewpoint). The number indicating the output order of the decoded image).

  FIG. 12 shows an example in which the signals at each viewpoint are interleaved in units of a plurality of images. In the figure, m (v, d) represents an image constituting each viewpoint image as in FIG. 11, v is a viewpoint number for specifying the viewpoint, and d is a number indicating the image capturing order at each viewpoint ( The number indicating the output order of the decoded image at each viewpoint).

  In the multi-view image encoding apparatus of FIG. 1, even when an image signal is input by any of the methods in FIGS. 5 to 12, it is input to the rearrangement buffer 103 and stored at any time without delay. Further, the image signals stored in the rearrangement buffer 103 are supplied to the motion / disparity compensation prediction unit 104 and the residual signal calculation unit 106 in units of pixel blocks according to the encoding order controlled by the encoding management unit 101. (Step S106 in FIG. 2).

  The encoding order controlled by the encoding management unit 101 will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of the encoding order of each image m (v, d) constituting a multi-view image of five viewpoints (V = 5) and a reference relationship of motion compensation / parallax compensation. In the figure, viewpoint images M (0), M (2), and M (4) are encoded without performing disparity compensation prediction referring to images of other viewpoints. For example, the image m (0,0) of the viewpoint image M (0) is encoded as a picture that is encoded independently only within the screen without referring to other images.

  The image m (0,3) of the viewpoint image M (0) is encoded using the motion compensated prediction using the decoded image of the front image m (0,0) in the display order of the same viewpoint as a reference image. Further, the image m (0, 1) of the viewpoint image M (0) is a reference image that is a decoded image of the front image m (0, 0) and the rear image m (0, 3) in the same viewpoint display order. Encode using motion compensated prediction.

  On the other hand, viewpoint images M (1) and M (3) are encoded using disparity compensation prediction that predicts an image of another viewpoint as a reference image in addition to motion compensation prediction. For example, the image m (1,1) of the viewpoint image M (1) is a reference image that is a decoded image of the front image m (1,0) and the rear image m (1,3) in the display order of the same viewpoint. In addition to performing motion compensation prediction, the decoded images of the images m (0, 1) and m (2, 1) of different viewpoints are used as reference images and encoded using parallax compensation prediction.

  When the image m (1, 1) of the viewpoint image M (1) is encoded, the images m (1, 0), m (1, 3), m (0, 1), and m ( 2, 1) must be encoded and decoded and stored in the decoded image buffer 110. In this example, m (0,0), m (2,0), m (1,0), m (4,0), m (3,0), m (0,3), m (2, 3), m (1,3), m (4,3), m (3,3), m (0,1), m (2,1), m (1,1), m (4,1 ), M (3,1), m (0,2), m (2,2), m (1,2), m (4,2), m (3,2), m (0,6) , M (2,6), m (1,6), m (4,6), m (3,6), m (0,4), m (2,4), m (1,4), It suffices to code in the order of encoding. The encoding order number c indicating the encoding order has a one-to-one correspondence with the decoded image output order number o and is managed by the encoding management unit 101.

  The motion / disparity compensation prediction unit 104 performs the above-described disparity compensation prediction in addition to performing the motion compensation prediction in the same manner as in the conventional AVC / H.264 method (step S107 in FIG. 2). In motion compensated prediction, an image of the same viewpoint in front or rear in the display order is used as a reference image, but in parallax compensated prediction, a common process can be performed if an image of another viewpoint is used as a reference image. In accordance with control of the encoding management unit 101, block matching is performed between the pixel block supplied from the rearrangement buffer 103 and the reference image supplied from the decoded image buffer 110. In the case of motion compensation prediction, a motion vector is used. In the case of disparity compensation prediction, a disparity vector is detected, a motion compensation prediction / disparity compensation prediction block signal is generated, and the motion compensation prediction / disparity compensation prediction block signal and the motion vector / disparity vector are sent to the encoding mode determination unit 105. Supply.

  The encoding management unit 101 controls candidate combinations such as whether to perform motion compensation prediction / disparity compensation prediction, the number of reference images, which decoded image to use as a reference image, and the size of a pixel block. Accordingly, motion compensation prediction / disparity compensation prediction is performed for all combinations that are candidates for all coding modes related to motion compensation prediction / disparity compensation prediction, and each motion compensation prediction / disparity compensation prediction block signal and motion vector / disparity are predicted. The vector is supplied to the encoding mode determination unit 105. The candidate pixel block size here is each small block obtained by further dividing the pixel block. For example, if the pixel block is 16 pixels in the horizontal direction and 16 pixels in the vertical direction (that is, 16 × 16), 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, 4 × 4, etc. Dividing into small blocks, motion compensation prediction is performed, and candidates are set.

  The encoding mode determination unit 105 determines which method of motion compensation prediction and disparity compensation prediction is selected and combined in which pixel block unit using which reference image, and efficient encoding can be realized. And the obtained encoding mode and the motion vector / disparity vector are supplied to the encoded bit string generation unit 111, and the motion compensation prediction / disparity compensation prediction block signal is supplied to the residual signal calculation unit 106. Supply (step S108 in FIG. 2).

  For example, when combining motion compensated prediction from the previous and subsequent reference images on the time axis, the motion compensated prediction block obtained by performing the motion compensated prediction from the previous reference image and the motion compensated prediction from the subsequent reference image are used. A block obtained by averaging the pixel values of the motion compensated prediction block obtained by the above is generated and set as a candidate. Also, motion compensation prediction and parallax compensation prediction can be combined. Furthermore, when averaging pixel values, weighting such as 1: 2 or 1: 3 may be used in addition to the average of 1: 1. Further, when the pixel block is divided into small blocks of 4 × 4 pixels to 16 × 16 pixels to be candidates for the encoding mode, the prediction method of each small block can be changed.

  There are various methods for determining the encoding mode. For example, there is a method for calculating the code amount and the distortion amount for each encoding mode and selecting the optimum encoding mode in the balance between the code amount and the distortion amount. is there. In this encoding mode determination, first, a residual signal is calculated for each combination of encoding modes, and the bit length of an encoded sequence obtained by encoding the residual signal, vector, and encoding mode is calculated. Code amount. Further, the encoded residual signal is decoded, and an absolute value error sum or a square sum of the decoded signal added with the prediction signal and the image signal before encoding is calculated to obtain the distortion amount. The code amount is multiplied by a predetermined multiplier and added to the distortion amount to obtain an evaluation value. The smallest evaluation value of the combinations of all candidate encoding modes is selected and set as the encoding mode of the pixel block.

  The residual signal calculation unit 106 subtracts the motion compensation prediction / disparity compensation prediction block signal supplied from the coding mode determination unit 105 from the pixel block signal supplied from the rearrangement buffer 103 to obtain a residual signal ( Step S109 in FIG. The residual signal encoding unit 107 performs a residual signal encoding process such as orthogonal transform and quantization on the residual signal input from the residual signal calculating unit 106 to calculate an encoded residual signal ( Step S110 in FIG.

  The encoding management unit 101 manages whether or not the encoded image is used as a reference image for motion compensation prediction of an image that follows in the encoding order or parallax compensation prediction of another viewpoint (FIG. 2). Step S111), when used as a reference image, the encoded residual signal is decoded, and the decoded image signal is sequentially stored in the decoded image buffer 110 in units of pixel blocks (steps S112 to S114 in FIG. 2).

  First, the residual signal decoding unit 108 performs a residual signal decoding process such as inverse quantization and inverse orthogonal transform on the encoded residual signal input from the residual signal encoding unit 107, thereby obtaining a decoded residual. A signal is generated (step S112 in FIG. 2). The residual signal superimposing unit 109 superimposes the decoded residual signal supplied from the residual signal decoding unit 108 on the prediction signal supplied from the encoding mode determining unit 105 to calculate a decoded image signal (step of FIG. 2). In step S113, the decoded image signal is sequentially stored in the decoded image buffer 110 in units of pixel blocks (step S114 in FIG. 2). The decoded image signal stored in the decoded image buffer 110 becomes a reference image for motion-compensated prediction of an image that follows in the coding order or parallax-compensated prediction of another viewpoint, if necessary.

  The encoded bit string generation unit 111 performs Huffman coding on the encoding mode input from the encoding mode determination unit 105, the motion vector or the disparity vector, the encoded residual signal input from the residual signal encoding unit 107, and the like. Encoding is performed sequentially using entropy encoding such as encoding and arithmetic encoding to generate an encoded bit string (step S115 in FIG. 2).

  As described above, the processing from step S106 to step S115 in FIG. 2 is repeated in units of pixel blocks until encoding of all the pixel blocks in the encoded image is completed (steps S105 to S116 in FIG. 2). Further, the processing from step S103 to step S116 in FIG. 2 is repeated for each encoded image (steps S102 to S117 in FIG. 2).

  Next, the multi-viewpoint image decoding apparatus of the present invention will be described with reference to the drawings. FIG. 14 is a block diagram of an embodiment of the multi-view image decoding apparatus according to the present invention, and FIG. 15 is a flowchart for explaining the processing procedure of the multi-view image decoding apparatus of FIG. As illustrated in FIG. 14, the multi-view image decoding apparatus according to the present embodiment includes an encoded bit sequence decoding unit 201, a decoded image management information calculation unit 202, a motion / disparity compensation prediction unit 203, a prediction signal synthesis unit 204, a residual, A signal decoding unit 205, a residual signal superimposing unit 206, a decoded image buffer 207, a decoded image management unit 208, and a decoded image output unit 209 are provided.

  The operation of the multi-view image decoding apparatus according to the present embodiment will be described with reference to the flowchart of FIG. First, the encoded bit sequence decoding unit 201 decodes an encoded bit sequence encoded by entropy encoding such as Huffman encoding or arithmetic encoding by the multi-view image encoding apparatus shown in FIG. Camera parameter information such as the number of viewpoints V and the viewpoint position of the camera is obtained (step S201 in FIG. 15). The camera parameter information is information necessary for displaying the decoded multi-viewpoint image on a multi-viewpoint image display device or the like serving as an output destination. The number of viewpoints V is also used when calculating the viewpoint number v that identifies the viewpoints of the images constituting the multi-viewpoint image, and the number d that indicates the output order of the decoded images at each viewpoint.

  Also, the encoded bit string decoding unit 201 decodes a decoded image necessary for calculating a viewpoint number v that identifies the viewpoint of each image constituting the multi-viewpoint image and a number d that indicates the output order of the decoded image at each viewpoint. An output order number o is obtained (step S203 in FIG. 15).

  The decoded image management information calculation unit 202 determines the viewpoint number v (provided that v is the number of viewpoints) that satisfies the expression (1) from the number V of viewpoints supplied from the encoded bit string decoding unit 201 and the decoded image output order number o. Is an integer greater than or equal to 0 and less than V) and a number d (integer) indicating the output order of the decoded image at each viewpoint is calculated (step S204 in FIG. 15). Specifically, the number d is a quotient obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. The number v is a remainder value obtained by dividing the decoded image output order number o by the number of viewpoints V by integer arithmetic. Alternatively, the viewpoint number v may be calculated by the equation (2) after calculating the number d.

  The number of viewpoints V supplied from the encoded bit string decoding unit 201, the decoded image output order number o, the viewpoint number v specifying the viewpoint calculated by the decoded image management information calculation unit 202, and the output of the decoded image at each viewpoint The number d indicating the order is supplied to a decoded image management unit 208 (to be described later) and used for management of the decoded image stored in the decoded image buffer 207.

  Here, in the multi-view image decoding apparatus according to the present embodiment, as in the case of encoding, when the conventional single-view decoding method is expanded as a multi-view decoding method, the decoded image output sequence number of this method is used. By treating o as a number indicating the output order of the decoded image of the conventional single-view decoding method, compatibility with the conventional decoding method can be achieved.

  For example, when the AVC / H.264 system is expanded to the multi-viewpoint encoding system, the decoded image output order number o of this system is a picture / number indicating the output order of the decoded image of the AVC / H.264 system. Treat as an order count. In addition, since the multi-view image decoding apparatus according to the present embodiment performs decoding in the order encoded on the encoding side, the encoding order is equal to the decoding order. Further, the encoded bit string decoding unit 201 obtains information such as the encoding mode, motion vector or disparity vector, encoded residual signal (encoded prediction residual signal) of the pixel block to be decoded (step in FIG. 15). S206).

  Subsequently, the motion / disparity compensation prediction unit 203 performs motion compensation prediction / disparity compensation prediction according to the coding mode decoded by the coded bit string decoding unit 201 (step S207 in FIG. 15). In this motion compensation prediction / parallax compensation prediction, an image supplied from the decoded image buffer 207 according to the encoding mode is referred to, and the pixel block at the position indicated by the motion vector / disparity vector decoded by the encoded bit string decoding unit 201 Is a motion compensation prediction / parallax compensation prediction block. The size of the pixel block is divided into small blocks, and the prediction method and motion vector / disparity vector of each small block may be different. In some cases, prediction is performed from a plurality of reference pictures. In such a case, a plurality of motion compensation predictions / disparity compensation predictions are performed to obtain a plurality of prediction blocks.

  The prediction signal synthesis unit 204 synthesizes a plurality of prediction blocks when the pixel block is divided into small blocks or is predicted from a plurality of reference pictures, and generates a prediction signal of the pixel block ( Step S208 in FIG. 15). On the other hand, the residual signal decoding unit 205 performs a residual signal decoding process such as inverse quantization and inverse orthogonal transform on the encoded residual signal input from the encoded bit string decoding unit 201, thereby obtaining a decoded residual signal. Is generated (step S209 in FIG. 15).

  The residual signal superimposing unit 206 calculates a decoded image signal by superimposing the decoded residual signal supplied from the residual signal decoding unit 205 on the prediction signal supplied from the prediction signal combining unit 204 (step in FIG. 15). In step S210, the decoded image signal is sequentially stored in the decoded image buffer 207 in units of pixel blocks (step S211 in FIG. 15). The decoded image signal stored in the decoded image buffer 207 becomes a reference image when decoding subsequent images in the encoding order, if necessary.

  The processing from step S206 to step S211 in FIG. 15 is repeated until decoding of all pixel blocks in the encoded image is completed in units of pixel blocks (steps S205 to S212 in FIG. 15).

  The decoded image management unit 208 outputs the number of viewpoints V supplied from the decoded image management information calculation unit 202, the decoded image output order number o, the viewpoint number v that identifies the viewpoint of each image, and the output of the decoded image at each viewpoint. The number d indicating the order and the decoded image signal stored in the decoded image buffer 207 are managed in association with each other. Based on these parameters, the decoded image management unit 208 determines whether to output the decoded image stored in the decoded image buffer 207 (step S213 in FIG. 15).

  When the decoding order and the output order of the decoded image are different and the output timing is different, a delay is necessary, and the decoded image may not be output. The decoded image management unit 208 specifies the viewpoint for each decoded image signal stored in the decoded image buffer 207 by the number v, manages the output order of the decoded image at each viewpoint by the number d, and manages each viewpoint. Control is performed so that the viewpoints are synchronized with each other so that images with the same value of the number d of the decoded image signals are output simultaneously or continuously, and output in ascending order of the value of the number d. The decoded image output unit 209 outputs the decoded image stored in the decoded image buffer 207 to the multi-view image display device or the like in synchronization with the decoded image signal of each viewpoint in accordance with the control of the decoded image management unit 208 ( Step S214 in FIG. 15).

  As a method of outputting the images of each viewpoint of the decoded viewpoint image M (v) in synchronization with each other, a method of outputting the image signals of each viewpoint in parallel on independent channels, and an interleaving of the image signals of each viewpoint There is a method of serially outputting with one channel. When the decoded image output unit 209 outputs the image signals of each viewpoint in parallel through independent channels, the image signal at the same time of each viewpoint, that is, the number d indicating the output order of the decoded images at each viewpoint is The decoded image signals of the same viewpoints are output in synchronization with each other (steps S213 and S214 in FIG. 15).

If it demonstrates using FIG. 3 used by description of the above-mentioned multiview image coding apparatus, about each viewpoint image M (v), the value of the number d of each image m (v, d) is an order from the smallest. By outputting the decoded images, the decoded images can be output in a desirable order when displayed on an output destination multi-viewpoint image display device or the like. Further, by outputting the decoded image signals having the same value of the number d at the same time, the decoded image signals of the respective viewpoints can be synchronized with each other. In that case, the decoding image of each viewpoint can be output without missing by starting the output of the decoding image after the images of all the viewpoints are decoded. (Steps S213 and S214 in FIG. 15)
In the case where the decoded image output unit 209 serially outputs each viewpoint as a signal interleaved with one channel, the number d indicating the output order of the decoded image at each viewpoint is ordered in ascending order. Images at the same time, that is, the decoded image signals of the respective viewpoints having the same number d indicating the output order of the decoded images at the respective viewpoints, are synchronized with each other and output. As a method of serially outputting each viewpoint as an interleaved signal, a method of interleaving each viewpoint signal in units of pixels, a method of interleaving in units of a plurality of pixels, a method of interleaving in units of horizontal lines, There are a method of interleaving in units of images, a method of interleaving in units of a plurality of images, and the like.

  The interleave structure of the decoded image signal to be output is the same as the interleave structure of the viewpoint image input to the above-described multi-view image encoding apparatus shown in FIGS. As in the case of outputting in parallel on independent channels, the method of interleaving each viewpoint signal in units of pixels, the method of interleaving in units of a plurality of pixels, and the method of interleaving in units of horizontal lines, for example, As shown in FIG. 5 to FIG. 10, the image signals of the respective viewpoints having the same number d are interleaved and output in units of pixels, units in which a plurality of pixels are grouped, and horizontal line units. Image signals can be output at the same time, and the viewpoints can be output in synchronization with each other.

  At this time, by starting the interleaving and output of the decoded image after the images of all the viewpoints to be interleaved are decoded, the decoded image of each viewpoint can be output without being lost (FIG. 15). Steps S213 and S214). Further, in the method of interleaving in units of images, as shown in FIG. 11, by sequentially outputting the image signals of the respective viewpoints having the same number d in units of images, the viewpoints can be output in synchronization with each other. it can. Further, the processing from step S203 to step S214 in FIG. 15 is repeated for each encoded image (steps S202 to S215 in FIG. 15).

  As described above, in the multi-view image encoding apparatus shown in FIG. 1, a method in which the respective viewpoints are input in parallel through independent channels, and a serial input of the respective viewpoints as interleaved signals through one channel. In any of the methods, the data is input to the rearrangement buffer 103 and stored without delay. Therefore, unlike the conventional stereoscopic image encoding method and apparatus, it is not necessary to sequentially arrange one frame or one field before encoding, and there is no image buffer for sequentialization, and the delay time is shortened. The effect that it can be done can be obtained.

  On the other hand, in the multi-view image decoding device of the present embodiment, when outputting the decoded image stored in the decoded image buffer 207 to the multi-view image display device or the like, the format according to the input of the display device or the like, The viewpoint is output by a method of outputting in parallel on independent channels or a method of serially outputting on one channel as a signal in which the viewpoints are interleaved. Therefore, unlike the conventional stereoscopic image decoding method and apparatus, there is no need to synchronize after decoding by sequentially arranging every frame or field after decoding, and there is no image buffer for synchronization. The effect that the delay time can be shortened can be obtained.

  In addition, in the multi-view image encoding apparatus of FIG. 1, the view number v that specifies the viewpoint of each image that constitutes the multi-view image signal and the number d that indicates the output order of the decoded image at each viewpoint are displayed as the decoded image. The decoded image output order number o collectively shown as the output order number o is encoded together with the number V of viewpoints of the multi-view image signal. On the other hand, in the multi-view image decoding apparatus according to the present embodiment, it is possible to reliably decode the encoded data encoded in this way, and to specify the viewpoint of each viewpoint image constituting the multi-view image signal. According to the number v and the number d indicating the output order of the decoded image at each viewpoint, it can be appropriately output to a multi-viewpoint image display device or the like.

  When the conventional single-viewpoint image encoding / decoding system is expanded to the multi-viewpoint image encoding / decoding system of the present embodiment, the viewpoint number v for identifying the viewpoint of each viewpoint image of this system and each viewpoint The decoded image output order number o that collectively indicates the output number d indicating the output order of the decoded images in FIG. 5 is the number that indicates the output order of the decoded images of the conventional single viewpoint image encoding / decoding method (for example, AVC / H. H.264 picture order count) and encoding / decoding can achieve an effect that compatibility with a conventional single-viewpoint image encoding / decoding method can be achieved with a small improvement.

  Note that the present invention is not limited to the above embodiment. For example, in the multi-view image decoding apparatus according to the present embodiment, the decoded image stored in the decoded image buffer 207 is output when the image is output. When rearrangement of pixels is necessary when outputting to a multi-viewpoint image display device or the like, a line buffer for temporary storage is provided inside or outside the decoded image output unit 209, and the decoded image read from the decoded image buffer 207 After the signal is temporarily written in the line buffer, the pixels are rearranged and output at any time, or the decoded image signal read from the decoded image buffer 207 is rearranged and temporarily written in the line buffer and then output at any time. This is also included in the present invention.

  Further, in the multi-view image encoding apparatus of FIG. 1, when inputting an image, the image is input to the rearrangement buffer 103 and stored without delay, but the input image format and the rearrangement buffer 103 If the pixels need to be rearranged when stored in the rearrangement buffer 103, such as in different storage formats, a line buffer for temporary storage is provided, and the input image signal is temporarily written in the line buffer, and then the pixels are rearranged as needed. The data can be stored in the rearrangement buffer 103, or the pixels can be rearranged and temporarily written in the line buffer, and then stored in the rearrangement buffer 103 as needed.

  Further, in the above description of the multi-view image decoding apparatus according to the present embodiment, the decoded image output unit 209 decodes the decoded image stored in the decoded image buffer 207 with the viewpoint number V and the decoded image management information calculation unit 202. Each viewpoint decoded according to the control of the decoded image management unit 208 based on the viewpoint number v specifying the viewpoint calculated from the image output order number o and the number d indicating the output order of the decoded image at each viewpoint. In the case where the viewpoints are interleaved in units of images and serially output in one channel, the image signals can be output based on the decoded image output order number o. In this case, by outputting the decoded images in order from the smallest decoded image output order number o, the viewpoints can be interleaved in units of images and output in synchronization with each other.

  In the above description, multi-viewpoint images used for encoding and decoding can be encoded and decoded from multi-viewpoint images actually captured from different viewpoints. A viewpoint image that has been converted or generated, such as interpolating a position from a peripheral viewpoint, can be encoded and decoded, and is included in the present invention. In addition, multi-view images such as computer graphics can be encoded and decoded, and is included in the present invention.

  For example, a multi-viewpoint image signal including image signals of four viewpoints A, B, C, and D is (1) an image signal obtained by actually photographing all four viewpoint image signals at each viewpoint. In some cases, (2) when all four viewpoint image signals are virtually taken at each viewpoint, (3) A and B viewpoint image signals are actually captured at each viewpoint. The image signal obtained by actually shooting and the image signal obtained by actually shooting, such as the image signal obtained by virtually capturing the image signal of the C and D viewpoints and the image signals of the C and D viewpoints. There are three cases where the generated image signal and the generated image signal are mixed.

  In addition, the position of each viewpoint of the multi-viewpoint image used in the present invention may be any arrangement. This will be described with reference to FIGS. The numbers shown in FIGS. 16 to 19 are viewpoint numbers v (v = 0, 1, 2,...) Indicating viewpoint positions. FIG. 16 shows an example in which the viewpoints are arranged in the horizontal direction. The images were taken with the cameras arranged horizontally. FIG. 17 shows an example in which the viewpoints are arranged in the vertical direction. The picture was taken with the cameras arranged vertically. 18 and 19 are examples in which the viewpoints are arranged in a two-dimensional horizontal / vertical direction. The images were taken with the cameras arranged in two horizontal / vertical directions. The value of the viewpoint number v that identifies the viewpoint has a one-to-one correspondence with each viewpoint and must be assigned an integer that is greater than or equal to 0 and less than the number of viewpoints V. However, there is consistency between the encoding side and the decoding side in terms of camera parameters. Any order is acceptable.

  The above multi-view image encoding and decoding processes can be realized as a transmission, storage, and reception device using hardware, as well as in a ROM (Read Only Memory), a flash memory, or the like. It can also be realized by stored firmware or software such as a computer. The firmware program and software program can be recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as a data broadcast of terrestrial or satellite digital broadcasting Is also possible.

It is a block diagram of an example of the multiview image coding apparatus which produces | generates the encoding bit sequence decoded by this invention decoding apparatus. 3 is a flowchart for explaining multi-view image encoding processing in FIG. 1. It is a figure explaining an example at the time of assigning a value to the viewpoint number v which specifies the viewpoint of each image, and the number d which shows the output order of the decoded image in each viewpoint. It is a figure explaining an example at the time of assigning a value to decoding picture output order number o of each picture. It is a figure explaining an example at the time of interleaving the signal of each viewpoint per pixel. It is a figure explaining an example at the time of interleaving the signal of each viewpoint by the unit which put together the several pixel. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in pixel block units, such as 16x16 and 16x8 pixels. It is a figure explaining an example at the time of interleaving the signal of each viewpoint by the line unit of a horizontal direction. It is a figure explaining an example at the time of interleaving in the form which put together the signal of each viewpoint in one picture. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in the slice unit which put together the some line. It is a figure explaining an example at the time of interleaving the signal of each viewpoint for every image. It is a figure explaining an example at the time of interleaving the signal of each viewpoint in the unit which put together the some image. It is a figure explaining an example of the encoding relationship of each image, and the reference relationship of motion compensation / parallax compensation. It is a block diagram of one embodiment of a multi-view image decoding device of the present invention. 15 is a flowchart for explaining the multi-view image decoding process of FIG. It is a figure explaining an example at the time of arrange | positioning a viewpoint to a horizontal direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint to a perpendicular direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint in a horizontal / vertical two-dimensional direction. It is a figure explaining an example at the time of arrange | positioning a viewpoint in a horizontal / vertical two-dimensional direction. It is a block diagram of an example of the conventional stereoscopic vision multi-view image encoding apparatus. It is a block diagram of an example of the conventional stereoscopic vision image decoding apparatus.

Explanation of symbols

101 Coding management unit
102 Decoded image output order number calculation unit
DESCRIPTION OF SYMBOLS 103 Rearrangement buffer 104 Motion / disparity compensation prediction part 105 Coding mode determination part 106 Residual signal calculation part 107 Residual signal encoding part 108 Residual signal decoding part 109 Residual signal superimposition part 110 Decoded image buffer 111 Encoding bit sequence Generation unit 201 Encoded bit stream decoding unit 202 Decoded image management information calculation unit 203 Motion / disparity compensation prediction unit 204 Prediction signal synthesis unit 205 Residual signal decoding unit 206 Residual signal superimposing unit 207 Decoded image buffer 208 Decoded image management unit 209 Decoding Image output unit

Claims (3)

It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image decoding device that decodes encoded data obtained by encoding a multi-view image signal that is an image signal generated as a virtual image taken from one viewpoint,
Decoding means for decoding the encoded data and generating a decoded multi-viewpoint image signal;
Storage means for storing the decoded multi-viewpoint image signal at any time in an image buffer;
A multi-viewpoint, comprising: an output unit that extracts a decoded multi-viewpoint image signal from the image buffer and outputs the decoded image signals of the respective viewpoints as independent channels for each decoded image signal of the respective viewpoints. Image decoding device. It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image decoding device that decodes encoded data obtained by encoding a multi-view image signal that is an image signal generated as a virtual image taken from one viewpoint,
Decoding means for decoding the encoded data and generating a decoded multi-viewpoint image signal;
Storage means for storing the decoded multi-viewpoint image signal at any time in an image buffer;
Output means for extracting a decoded multi-viewpoint image signal from the image buffer, synchronizing the decoded image signals of the respective viewpoints with each other, interleaving the decoded image signals of the respective viewpoints, and outputting them serially in one channel, Multi-viewpoint image decoding apparatus. It is a multi-viewpoint image signal including image signals of each viewpoint obtained respectively at a plurality of set viewpoints, and the image signal of one viewpoint is an image signal obtained by actually photographing from the one viewpoint, or A multi-view image decoding device that decodes encoded data obtained by encoding a multi-view image signal that is an image signal generated as a virtual image taken from one viewpoint,
A decoded image that collectively indicates the multi-view image signal, the number V of viewpoints of the multi-view image signal, a number v that identifies each viewpoint, and a number d that indicates the output order of the decoded image at each viewpoint The encoded data in which the output order number o (integer) is encoded is decoded, and the decoded multi-view image signal, the viewpoint number V of the multi-view image signal, the decoded image output order number o, and Each of the decoding means for generating
A quotient (integer) obtained by dividing the decoded image output order number o by the integer operation by the number V of the decoded viewpoints is calculated as a number d indicating the output order of the decoded images at each viewpoint. And calculating means for calculating the remainder of the division (an integer of 0 or more and less than V) as a number v for identifying each of the viewpoints;
Storage means for storing the decoded multi-viewpoint image signal in an image buffer;
In accordance with a number d indicating the output order of the decoded images at the respective viewpoints supplied from the calculation means and a number v specifying each of the viewpoints, a decoded multi-viewpoint image signal is extracted from the image buffer, A multi-view image decoding apparatus comprising: output means for outputting the decoded image signals of the respective viewpoints constituting the decoded multi-view image signal in synchronization with each other.

Images