JP2010157824A - Image encoder, image encoding method, and program of the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To transmit or accumulate multi-viewpoint images efficiently. <P>SOLUTION: An image signal encoder 107 encodes a plurality of images respectively made from a plurality of different visual points to generate image encoded data. A depth information encoder (for example, a depth signal encoder 108) encodes depth information indicative of a depth of a specific space from at least one or more visual points to generate depth information encoded data. A parameter information encoder 110 encodes parameter information including visual point information for specifying a plurality of visual points, a basis of the plurality of images and the depth information, to generate parameter information encoded data. A unitization portion 109 generates an encoded stream containing the image encoded data generated by the image signal encoder 107, the depth information encoded data generated by the depth information encoder, and the parameter information encoded data generated by the parameter information encoder 110. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image encoding device, an image encoding method, and a program thereof that encode images captured from a plurality of different viewpoints.

  In recent years, applications using images from multiple viewpoints have become widespread. One such application is a twin-lens stereoscopic television. In a twin-lens stereoscopic television, a left-eye image and a right-eye image, which are taken from two different directions by two cameras, are generated and displayed on the same screen to show a stereoscopic image. ing. In this case, the left-eye image and the right-eye image are separately transmitted or recorded as independent images. In this case, the amount of information about twice that of a single two-dimensional image is required.

  Therefore, a method has been proposed in which one of the left and right images is set as a main image, the other image is set as a sub image, and information of the sub image is information-compressed by a general compression encoding method to reduce the amount of information ( For example, see Patent Document 1). In this proposed stereoscopic television image transmission method, a relative position having a high correlation with the main image is obtained for each small area of the sub-image, and its position deviation amount (hereinafter referred to as a disparity vector) and a difference signal (hereinafter referred to as a disparity vector). The prediction residual signal) is transmitted or recorded. If the main image and the disparity vector are used, an image close to the sub-image can be restored, but the prediction residual signal is also transmitted or recorded because the information of the sub-image that does not have the main image such as a shadow part of the object cannot be restored Because.

  In 1996, a stereo image encoding method called a multi-view profile was added to the MPEG-2 video (ISO / IEC 13818-2) encoding method, which is an international standard for single-view image encoding (ISO). / IEC 13818-2 / AMD3). The MPEG-2 video multi-view profile is a two-layer encoding method that encodes the image for the left eye with the basic layer and the image for the right eye with the enhancement layer, and motion using redundancy in the time direction In addition to compensation prediction and discrete cosine transformation using redundancy in the spatial direction, compression coding is performed using disparity compensation prediction using redundancy between viewpoints.

  In addition, a technique has been proposed for reducing the amount of information for multi-viewpoint images captured by three or more cameras using motion compensation prediction and parallax compensation prediction (see, for example, Patent Document 2). This proposed image high-efficiency coding method performs pattern matching with reference pictures of multiple viewpoints, and selects a motion-compensated prediction image or a parallax-compensated prediction image that minimizes an error, thereby improving the coding efficiency. Has improved.

In JVT (Joint Video Team), AVC / H. The standardization work of a multi-view image coding method (MVC: Multiview Video Coding (hereinafter referred to as MVC method)) in which the H.264 coding method (see Non-Patent Document 1) is extended to a multi-view image is progressing (Non-Patent Document 1). 2). Similar to the MPEG-2 video multi-view profile described above, the MVC method also improves the coding efficiency by incorporating prediction between viewpoints.
JP-A 61-144191 JP-A-6-98312 ITU-T Recommendation H.264 (11/2007) Joint Draft 6.0 on Multiview Video Coding, Joint Video Team of ISO / IEC MPEG & ITU-T VCEG, JVT-Z209, January 2008

  A multi-view image from a plurality of viewpoints can be encoded using the various methods described above. However, these methods encode all images of the necessary viewpoints, and considering the limited transmission speed and storage capacity, it is difficult to efficiently transmit or store multi-viewpoint images. Many. For example, if a large number of viewpoints are required, the amount of data will become very large if all the images from those viewpoints are transmitted or stored. That is, a very large amount of data must be received or read on the decoding side. Also, it is difficult to generate a free viewpoint image according to a user instruction on the decoding side with high accuracy.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an image encoding device, an image encoding method, and a program thereof that can efficiently transmit or store multi-viewpoint images.

  An image encoding device according to an aspect of the present invention includes a first encoding unit that encodes a plurality of images from a plurality of different viewpoints to generate encoded image data, and identification from at least one or more viewpoints. A second encoding unit that encodes depth information indicating the depth of the space to generate depth information encoded data, and includes viewpoint information for specifying a plurality of viewpoints based on a plurality of images and depth information A third encoding unit that encodes parameter information and generates parameter information encoded data; and image encoded data generated by the first encoding unit, the second encoding unit, and the third encoding unit, A stream generation unit that generates an encoded stream including the depth information encoded data and the parameter information encoded data.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, multi-viewpoint images can be transmitted or stored efficiently.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, AVC / H. An example will be described in which a multi-view image is encoded using a method that is an extension of the MVC method in which the H.264 encoding method is extended to a multi-view image.

  First, AVC / H. The H.264 encoding method will be briefly described. AVC / H. The H.264 encoding method realizes higher encoding efficiency than conventional encoding methods such as MPEG-2 video (ISO / IEC 13818-2) and MPEG-4 visual (ISO / IEC 14496-2). Yes.

  In a P picture (that is, a forward prediction encoded image) in an encoding method such as MPEG-2 video or MPEG-4 visual, motion compensation prediction is performed only from the immediately preceding I picture or P picture in display order. In contrast, AVC / H. In the H.264 coding system, a plurality of P pictures and B pictures can be used as reference pictures, and motion compensation can be performed by selecting an optimum picture for each block. In addition to the preceding picture in the display order, a subsequent picture in the already encoded display order can also be referred to.

  A B picture in an encoding system such as MPEG-2 video or MPEG-4 visual refers to one reference picture in the display order, one reference picture in the rear, or two reference pictures at the same time. The average value of the two pictures is used as a reference picture for prediction, and difference data between the target picture and the reference picture is encoded. In contrast, AVC / H. In the H.264 coding system, a B picture is not restricted by the restriction of one front and one rear in the display order, and an arbitrary reference picture can be referred for prediction regardless of the front or rear. Further, a B picture can refer to a B picture other than itself as a reference picture.

  Furthermore, the encoding mode for each picture or VOP has been determined using a picture in MPEG-2 video and a video object plane (VOP) in MPEG-4 as one unit. In the H.264 encoding method, a slice is used as an encoding unit, and different slices such as an I slice, a P slice, and a B slice can be mixed in one picture.

  Furthermore, AVC / H. In the H.264 encoding method, a VCL (Video Coding Layer) for encoding or decoding a video pixel signal (that is, encoding mode, motion vector, DCT coefficient, etc.) and NAL (Network Abstraction Layer) ; Network abstraction layer) is defined.

  AVC / H. An encoded stream encoded by the H.264 encoding method is configured in units of NAL units that are one segment of NAL. The NAL unit includes a VCL NAL unit that includes VCL-encoded data (ie, encoding mode, motion vector, DCT coefficient, etc.) and a non-VCL NAL unit that does not include VCL-generated data. is there. The non-VCL NAL unit includes an SPS (Sequence Parameter Set) that includes parameter information related to encoding of the entire sequence, a PPS (Picture Parameter Set) that includes parameter information related to encoding of pictures, There is SEI (Supplemental Enhancement Information) which is not necessary for decoding data encoded by VCL.

  In addition, AVC / H. The basic unit of encoding in the H.264 encoding method is a slice in which a picture is divided, and the NAL unit of the VCL is a slice unit. Therefore, a unit called an access unit in which several NAL units are collected is defined, and one encoded picture is included in one access unit.

  Next, the MVC method will be briefly described. Here, the relationship between the viewpoints when encoding each image of the multi-view image and decoding the encoded stream, and the reference dependency relationship between the encoding target images constituting the multi-view image, This will be described using an example of five viewpoints.

  FIG. 2 is a diagram illustrating an example of a reference dependency relationship between images when a multi-view image including five viewpoints is encoded by the MVC method. The vertical axis indicates the spatial direction of a plurality of viewpoints (in this specification, the spatial direction of the plurality of viewpoints is the viewpoint direction), and the horizontal axis indicates the time direction of shooting or display order. P (v, t) (viewpoint v = 0, 1, 2,...; Time t = 0, 1, 2,...) Is an image of the viewpoint v at time t.

  In addition, the image indicated on the end point side of the arrow is a target picture to be encoded or decoded. The image pointed to by the start point of the arrow is a reference picture that is referred to when the target picture is encoded or decoded. That is, the reference picture is referred to in inter prediction (eg, motion compensation prediction) or inter-view prediction (eg, disparity compensation prediction) in the time direction. More specifically, the image pointed to by the start point of the horizontal arrow is a reference picture that is referred to by inter prediction in the temporal direction when the target picture is encoded or decoded, and the image of the vertical arrow The image pointed to by the start point side is a reference picture that is referred to in inter-view prediction.

  Here, inter prediction in the time direction is a prediction method that refers to an image at another time, and inter-view prediction is a prediction method that refers to an image at another viewpoint. In addition, the image used as the reference picture for inter prediction in the temporal direction is only the image preceding in the encoding or decoding order in the temporal direction, and the image used as the reference picture for inter-view prediction is encoded or decoded in the viewpoint direction. Only images that precede in order. For example, in the reference dependency relationship illustrated in FIG. 2, the viewpoint encoding or decoding order in the viewpoint direction may be set to viewpoint 0, viewpoint 2, viewpoint 1, viewpoint 4, and viewpoint 3. Further, the encoding or decoding order of viewpoints in the time direction may be t = 0, 4, 2, 1, 3, 8, 6, 5, 7,. First, P (0,0), P (2,0), P (1,0) are set at the same time in accordance with the viewpoint encoding or decoding order of viewpoints in the viewpoint direction. ), P (4,0), P (3,0) in this order. After that, the images of the respective viewpoints having t of 4 are similarly expressed in the order of encoding or decoding of the viewpoints in the viewpoint direction, and P (0, 4), P (2, 4), P (1, 4), P ( 4, 4) and P (3, 4) in this order. Hereinafter, the same processing is performed for images after the images of the respective viewpoints with t = 2.

  In addition, viewpoint 0 is set as a base viewpoint. In the MVC encoding method, the base viewpoint refers to a viewpoint that can be encoded or decoded without depending on other viewpoints. Only one viewpoint is the base viewpoint in the entire sequence of multi-viewpoint images. That is, the base viewpoint can be encoded or decoded independently without using an image of another viewpoint as a reference image for inter-view prediction. For non-base viewpoints (that is, viewpoints other than the base viewpoint), images from other viewpoints can be used as reference images for inter-view prediction.

  Furthermore, the MVC scheme has a mechanism for encoding the number of viewpoints of the multi-view image to be encoded, the encoding or decoding order in the viewpoint direction, and the reference dependency relationship between the viewpoints by inter-view prediction as a whole sequence. Yes. Encoding is performed by extending SPS which is a parameter set of sequence information.

  By encoding the above parameters, that is, the number of viewpoints and the viewpoint dependency information of each viewpoint, on the encoding side, the reference dependence relationship of each viewpoint can be determined on the decoding side as the entire sequence. The reference dependency information of each viewpoint is used for decoding processing such as initialization of a reference picture list for an inter-view prediction picture.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an image encoding device 100 according to Embodiment 1. The image encoding apparatus 100 according to Embodiment 1 includes an encoding management unit 101, a parameter information encoding unit 110, an image signal encoding unit 107, and a depth information encoding unit (more specifically, a depth signal encoding unit). 108). The parameter information encoding unit 110 includes an image signal sequence information encoding unit 102, a depth signal sequence information encoding unit 103, an image signal picture information encoding unit 104, a depth signal picture information encoding unit 105, and a camera parameter. An information encoding unit 106 is included.

  These configurations can be realized in hardware by any computer's CPU, memory, and other LSIs, and in software, they are realized by programs loaded into the memory. Draw functional blocks. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The image signal encoding unit 107 encodes a plurality of images from a plurality of different viewpoints, and generates encoded image data. The plurality of images may be images actually captured by a camera or images generated by computer graphics. When one viewpoint to be a reference among the plurality of viewpoints is set, the image signal encoding unit 107 encodes an image from the viewpoint to be the reference among the plurality of images and performs first image encoding. Data can be generated and other images can be encoded to generate second image encoded data.

  At that time, the image signal encoding unit 107 may perform intra-frame prediction encoding of an image from the viewpoint to be used as a reference among the plurality of images, and may perform inter-frame prediction encoding of the other images between the plurality of images. . In the MVC method, the viewpoint that should be the reference is the base viewpoint described above. The inter-frame prediction encoding is the inter-view prediction encoding described above.

  When the plurality of images are moving images, the image signal encoding unit 107 can also perform inter-frame predictive encoding of the moving images from the respective viewpoints in the time direction. Of course, inter-frame prediction encoding in the viewing direction and inter-frame prediction encoding in the temporal direction can be used in combination.

  The depth information encoding unit encodes depth information indicating the depth of the specific space from at least one viewpoint, and generates depth information encoded data. The depth information may be represented by an image in a monochrome format from a certain viewpoint (hereinafter referred to as a monochrome image as appropriate). In this case, the depth information encoding unit encodes the monochrome image to generate depth information encoded data.

  The depth information encoding unit performs intra-frame predictive encoding of a monochrome image from a viewpoint to be a reference among a plurality of monochrome images from a plurality of different viewpoints, and converts the other monochrome images between the plurality of monochrome images. Inter-frame predictive coding may be used. Here, the viewpoint that should be the basis of the monochrome image may coincide with the viewpoint that should be the basis of the image that is to be encoded by the image signal encoding unit 107, or may be different. It may be.

  In addition, when the plurality of monochrome images are moving images, the depth information encoding unit can also perform inter-frame predictive encoding of moving images in monochrome format from each viewpoint in the time direction. Of course, inter-frame prediction encoding in the viewing direction and inter-frame prediction encoding in the temporal direction can be used in combination.

  Here, the number of viewpoints from which the depth information is based may be set to be smaller than the number of viewpoints from which the first encoding unit is to be encoded, or the number of both viewpoints is May be set to match. In addition, the position of each viewpoint that is the basis of the depth information is set so as to coincide with any of the positions of the plurality of viewpoints that are the basis of the plurality of images to be encoded by the image signal encoding unit 107. It may be set so as not to match any of them.

  The parameter information encoding unit 110 generates parameter information encoded data by encoding parameter information including viewpoint information for specifying a plurality of viewpoints based on the plurality of images and the depth information. As described above, when one viewpoint to be used as a reference is set, the parameter information encoding unit 110 sets the first parameter information of an image from the viewpoint to be used as the reference among the plurality of images, and other information. The second parameter information of the image and the third parameter information of the depth information are encoded to generate first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data.

  Here, the third parameter information is described in a syntax structure corresponding to the syntax structure of the second parameter information. For example, the second parameter information and the third parameter information are stored in AVC / H. It can be described in accordance with the H.264 encoding multi-view high profile. Viewpoint identification information is described in the second parameter information and the third parameter information. When the viewpoint position that is the basis of the image to be encoded by the image signal encoding unit 107 matches the viewpoint position that is the basis of the depth information, common identification information is assigned to these viewpoints. The That is, the viewpoint identification information is managed uniformly between the image and the depth information.

  The unitization unit 109 generates an encoded stream including the image encoded data and the depth information encoded data generated by the image signal encoding unit 107 and the depth information encoding unit, respectively. The unitization unit 109 can also generate an encoded stream that further includes the parameter information encoded data generated by the parameter information encoding unit 110.

  When one viewpoint to be used as a reference is set among a plurality of viewpoints that are the basis of an image to be encoded by the image signal encoding unit 107, the unitization unit 109 includes the image signal encoding unit 107, The first image encoded data, the second image encoded data, the depth information encoded data, and the first parameter information encoded data generated by the depth information encoding unit and the parameter information encoding unit 110, respectively. Then, an encoded stream including the second parameter information encoded data and the third parameter information encoded data is generated.

  FIG. 3 is a block diagram showing a configuration of an image encoding device 100a according to a modification of the first embodiment. The image coding device 100a according to the modification of the first embodiment has a configuration in which a depth information generation unit (more specifically, a depth signal generation unit 111) is added to the image coding device 100 illustrated in FIG. .

  In the modification, the depth information generation unit generates depth information indicating the depth of a specific space from at least one viewpoint from a plurality of images to be encoded by the image signal encoding unit 107. The depth information generation unit can generate the depth information using an existing algorithm. The depth information encoding unit encodes the depth information generated by the depth information generation unit to generate depth information encoded data. The other processing is the same as the description of the image coding apparatus 100 according to the basic example of the first embodiment shown in FIG.

  Here, an image to be encoded by the image encoding apparatus 100 according to Embodiment 1 and depth information will be described. The image is a picture that can be obtained by projecting a subject onto a two-dimensional plane corresponding to each viewpoint by an imaging device such as a camera. The image signal is obtained by converting an image, which is two-dimensional information, into a one-dimensional signal flow. Note that the minimum unit of digitally expressed images and image signals is pixels. The multi-view image signal input to the image encoding apparatus 100 is a multi-view image signal including image signals respectively obtained from a plurality of set two or more viewpoints. The image signal of a certain viewpoint may be an image signal obtained by actually photographing from the viewpoint, or an image signal generated by computer graphics or the like as virtually photographed from the viewpoint. There may be. In addition, position correction and luminance / color level correction may be performed on an image signal obtained by actual shooting in order to correct variations in each camera used for the shooting.

  The depth signal may be a multi-view depth signal including depth signals respectively obtained from a plurality of set two or more viewpoints. The depth signal of a certain viewpoint may be a depth signal obtained by actually photographing from the viewpoint by an infrared camera or the like, or the above-mentioned multi-viewpoint image signal may be virtually captured from the viewpoint. It may be a depth signal originally generated by calculation.

  The depth information is information indicating the depth of the specific space. For example, the depth information is represented as depth information with respect to an image plane of a subject (that is, an object) in the image. More specifically, the depth information is information indicating the depth of the image projected on the two-dimensional plane. An image in which depth information corresponding to each pixel of an image projected on a two-dimensional plane is mapped in units of pixels or in units of a plurality of pixels is referred to as a depth map. The depth signal is obtained by converting a depth map, which is two-dimensional information, into a one-dimensional signal flow. Note that, as with images and image signals, the digitally expressed depth map and the minimum unit of depth signals are also pixels. The depth signal may be a multi-view depth signal including depth signals respectively obtained from a plurality of set two or more viewpoints. The depth signal of a certain viewpoint may be a depth signal obtained by actually photographing from the viewpoint by an infrared camera or the like, or the above-mentioned multi-viewpoint image signal may be virtually captured from the viewpoint. It may be a depth signal originally generated by calculation.

  The pixel value of the depth signal is often expressed by 8 bits like the image signal, but may be expressed by about 9 to 14 bits in order to improve the reproducibility in the depth direction. The depth map is represented as an image in monochrome format. Note that the resolution of the depth map may be set lower than the resolution of the image as long as the correspondence with each pixel of the image can be obtained.

  The depth signal is mainly used to generate an image signal of a desired virtual viewpoint that does not exist from an image signal of an existing viewpoint. When a free viewpoint image in which the viewpoint of an image to be displayed in accordance with a user instruction cannot be specified in advance is displayed on the decoding side, or there are many viewpoints, and all images from those viewpoints are captured, transmitted, or stored If this is difficult, it is effective to generate a virtual viewpoint image signal.

  One technique for generating a virtual viewpoint image that does not exist from an existing viewpoint image signal is disclosed in Japanese Patent Laid-Open No. 9-81746. In this method, when generating an image of a virtual viewpoint that does not exist, depth information is calculated from an image signal of an existing viewpoint, and a desired virtual viewpoint image is generated according to the depth information.

  Transmit or store the encoded stream obtained by encoding the multi-viewpoint image signal, obtain the depth signal from the image signal obtained by decoding the encoded stream, and generate the desired virtual viewpoint image signal This technique places a heavy burden on the decoding side to calculate the depth signal. In general, the quality of the depth signal generated on the decoding side is lower than the quality of the depth signal generated on the code side. This is because a high-frequency component of the original image signal is omitted in the general encoding method when encoding.

  Therefore, in the present embodiment, a depth signal is generated from a multi-viewpoint image signal on the encoding side, and a plurality of viewpoint image signals and a plurality of viewpoint depth signals are to be encoded. The decoding side can obtain a depth signal in addition to the image signal by decoding the encoded stream. Thereby, there is no need to generate a depth signal after decoding, and the decoding side can generate an image signal of a desired virtual viewpoint from the image signal and the depth signal obtained by decoding the encoded stream.

  When generating an image signal of a virtual viewpoint, it is better to generate an image from image signals and depth signals of a plurality of viewpoints than to generate an image from an image signal and a depth signal of one viewpoint. A virtual viewpoint image can be obtained. Hereinafter, this knowledge will be described in more detail with reference to FIGS. 4 and 5.

FIG. 4 shows a scene in which the first object OA and the second object OB exist from the second viewpoint VB and the third viewpoint VC, and the first viewpoint VA (hereinafter referred to as the first virtual viewpoint VA) which is a virtual viewpoint. It is a figure which shows the example which produces | generates the image of (notation).
FIG. 5 is a diagram illustrating a captured image, a corresponding depth map, and a generated image in the example of FIG. In FIG. 5, the second image IB shows an image taken from the second viewpoint VB in FIG. 4, and the third image IC shows an image taken from the third viewpoint VC in FIG. The second depth map DB indicates a depth map corresponding to the second image IB, and the third depth map DC indicates a depth map corresponding to the third image IC.

  In the following description, the pixel value of the depth signal corresponding to the rearmost object, that is, the object farthest from the camera is set to 0 which is the minimum value, and the closer the object is to the front, that is, the closer to the camera, The pixel value of the depth signal is set to a large value. Further, the first image IA indicates an image (hereinafter referred to as a predicted image) obtained on the assumption that the first image is taken from the first virtual viewpoint VA, and is not an actual image but an image to be generated.

  The first-second image IAB is a predicted image of the first virtual viewpoint VA generated from the second image IB photographed from the second viewpoint VB and the corresponding second depth map DB. When the predicted image of the first virtual viewpoint VA is generated from the second image IB photographed from the second viewpoint VB and the corresponding second depth map DB, the first object ahead when photographed from the second viewpoint VB The part concealed by OA is unknown and a missing part occurs. The black portion of the first-second image IAB is the second missing portion LPB that occurs in the predicted image of the first virtual viewpoint VA.

  The first-third image IAC is a predicted image of the first virtual viewpoint VA generated from the third image IC photographed from the third viewpoint VC and the corresponding third depth map DC. Missing portions also occur in the first-3 image IAC. The black portion of the first-3 image IAC is a third missing portion LPC that occurs in the predicted image of the first virtual viewpoint VA. The third missing portion LPC of the first-3 image IAC occurs at a position different from the second missing portion LPB of the 1-2 image IAB.

  Therefore, by supplementing the second missing portion LPB of the 1-2 image IAB with the image signal of the 1-3 image IAC, the first image IA of the first virtual viewpoint VA with few missing portions can be generated. Actually, the object has a three-dimensional effect or shadow, and brightness or color difference occurs in the image obtained by photographing depending on the relative relationship between the position and direction of the viewpoint to be photographed and the light source. FIG. 5 is drawn without considering that point.

  In order to take into account the luminance difference between the respective viewpoints and to reduce noise, the average value is used for pixels existing in both the 1-2 image IAB and the 1-3 image IAC, and one image is used. There is also a method in which only the pixel of the other image is used only for the pixel in which the missing portion occurs. Thus, the image signal of two viewpoints rather than the image of the virtual viewpoint generated from the image signal and the depth signal of one viewpoint (in FIG. 5, the first-second image IAB or the first-third image IAC). In addition, the image generated from the depth signal can obtain a good image with fewer missing portions.

  In addition, it is possible to generate a better image with fewer missing parts by using an image signal and a depth signal of more viewpoints than generating an image signal of a virtual viewpoint from the image signals and depth signals of two viewpoints. Obtainable. Thus, when generating an image of a virtual viewpoint, it is better to generate an image from image signals and depth signals of a plurality of viewpoints than to generate an image from an image signal and a depth signal of one viewpoint. A virtual viewpoint image can be obtained.

  In addition, when generating an image signal of a virtual viewpoint from an image signal and a depth signal of two viewpoints, the distance between the viewpoints is more likely to be generated from an image signal and a depth signal of two viewpoints having a short distance between the viewpoints. A better virtual viewpoint image signal generated from an image signal and a depth signal of two long viewpoints can be obtained. Hereinafter, this knowledge will be described in more detail with reference to FIGS.

FIG. 6 shows a scene where the third object OC and the fourth object OD exist from the fifth viewpoint VE and the sixth viewpoint VF, and a fourth viewpoint VD (hereinafter referred to as a fourth virtual viewpoint VD) which is a virtual viewpoint. It is a figure which shows the example which produces | generates the image of (notation).
FIG. 7 is a diagram illustrating a captured image, a corresponding depth map, and a generated image in the example of FIG. In FIG. 7, the fifth image IE shows an image taken from the fifth viewpoint VE in FIG. 6, and the sixth image IF shows an image taken from the sixth viewpoint VF in FIG. The fifth depth map DE indicates a depth map corresponding to the fifth image IE, and the sixth depth map DF indicates a depth map corresponding to the third image IC. The fourth image ID indicates a predicted image obtained when it is assumed that the image is captured from the fourth virtual viewpoint VD, and is not an actual image but an image to be generated.

  The 4-5th image IDE is a predicted image of the fourth virtual viewpoint VD generated from the fifth image IE photographed from the fifth viewpoint VE and the corresponding fifth depth map DE. When the predicted image of the fourth virtual viewpoint VD is generated from the fifth image IE photographed from the fifth viewpoint VE and the corresponding fifth depth map DE, the third object ahead is photographed from the fifth viewpoint VE. The part concealed by the OC is unknown and a missing part occurs. The black portion of the 4-5th image IDE is the fifth missing portion LPE that occurs in the predicted image of the fourth virtual viewpoint VD.

  The fourth to sixth image IDF is a predicted image of the fourth virtual viewpoint VD generated from the sixth image IF photographed from the sixth viewpoint VF and the corresponding sixth depth map DF. A missing part also occurs in the 4th-6th image IDF. The black portion of the 4-6th image IDF is the sixth missing portion LPF that occurs in the predicted image of the fourth virtual viewpoint VD.

  Comparing the fifth viewpoint VE and the sixth viewpoint VF, since the sixth viewpoint VF is farther from the fourth virtual viewpoint, the shift amount of the sixth image IF from the fourth image ID becomes larger. The area of the sixth missing portion LPF of the 4-6th image IDF is larger than the area of the fifth missing portion LPE of the 4-5th image IDE. As described above, the smaller the distance between the viewpoints, the smaller the shift amount, deformation, brightness, and color difference between the viewpoints of the subject in the image, and a good image can be obtained. Therefore, when generating an image signal of a virtual viewpoint, it is more preferable to generate an image signal and a depth signal of a plurality of viewpoints having a longer distance between the viewpoints by generating the image signal and the depth signal of the viewpoints having a shorter distance between the viewpoints. A better virtual viewpoint image generated from the image can be obtained.

  Also, the ease with which a virtual viewpoint image signal is generated varies depending on the depth state of the content. The smaller the difference in depth between overlapping subjects, the better the virtual viewpoint image signal can be obtained. Hereinafter, this knowledge will be described in more detail with reference to FIGS.

  FIG. 8 shows two scenes in which either the fifth object OE or the sixth object OF and the seventh object OG are present from the eighth viewpoint VH, and the seventh viewpoint VG ( Hereinafter, an example of generating an image of the seventh virtual viewpoint VG) will be described. At the time of shooting each scene, the fifth object OE and the sixth object OF do not exist at the same time. Here, a scene where the fifth object OE and the seventh object OG are present is a first scene H1, and a scene where the sixth object OF and the seventh object OG are present is a second scene H2.

  FIG. 9 is a diagram illustrating a captured image, a corresponding depth map, and a generated image in the example of FIG. In FIG. 9, an 8-1 image IH1 shows an image of the first scene H1 taken from the eighth viewpoint VH in FIG. 8, and an eighth-2 image IH2 is taken from the eighth viewpoint VH in FIG. An image of the second scene H2 is shown. The eighth-first depth map DH1 indicates a depth map corresponding to the eighth-first image IH1, and the eighth-second depth map DH2 indicates a depth map corresponding to the eighth-second image IH2.

  The seventh-first image IG1 is a predicted image obtained when it is assumed that the first scene H1 is photographed from the seventh virtual viewpoint VG, and is not actually photographed but an image to be generated. The seventh-second image IG2 is a predicted image obtained when it is assumed that the second scene H2 is captured from the seventh virtual viewpoint VG, and is not actually captured but is an image to be generated. .

  The seventh-8-1 image IGH1 is a seventh virtual viewpoint generated from the eighth-first image IH1 of the first scene H1 photographed from the eighth viewpoint VH and the corresponding eighth-first depth map DH1. It is a prediction image of the 1st scene H1 of VG. When generating a predicted image of the first scene H1 of the seventh virtual viewpoint VG from the 8-1 image IH1 of the first scene H1 photographed from the eighth viewpoint VH and the corresponding 8-1 depth map DH1, When the image is taken from the eight viewpoints VH, the portion hidden by the front fifth object OE is unknown, and a missing portion occurs. The black portion of the seventh-8-1st image IGH1 is the eighth-first missing portion LPH1 that occurs in the predicted image of the first scene H1 of the seventh virtual viewpoint VG.

  In addition, the seventh-8-2 image IGH2 is a seventh virtual viewpoint generated from the eighth-2 image IH2 of the second scene H2 taken from the eighth viewpoint VH and the corresponding eighth-2 depth map DH2. It is a prediction image of the 2nd scene H2 of VG. A missing portion is also generated in the 7-8-2 image IGH2. The blackened portion of the seventh-8-8 image IGH2 is the eighth-2 missing portion LPH2 generated in the predicted image of the second scene H2 of the seventh virtual viewpoint VD.

  When the shift amount between the 8-1st image IH1 and the 7-1th image IG1 is compared with the shift amount between the 8-2nd image IH2 and the 7th-2 image IG2, the latter becomes larger. Therefore, the area of the 8-2 missing portion LPH2 of the 7-8-2 image IGH2 is larger than the area of the 8-1 missing portion LPH1 of the 7-8-1 image IGH1. Thus, the ease of generating the image signal of the virtual viewpoint varies depending on the depth state of the content. In other words, the smaller the difference in depth between the overlapping subjects, the smaller the relative shift amount between the overlapping subjects in the image, and the smaller the missing portion of the generated image, the better the image. Can do.

  Note that the difference in depth between overlapping subjects can be calculated from the depth signal. The edge of the depth signal (the 8-1 depth map DH1 and the 8-2 depth map DH2 in FIG. 9) (that is, the point where the density changes sharply) is extracted, and the pixel value across the boundary of the edge portion is extracted. The difference is calculated, and the smaller the difference is, the smaller the difference in depth between the overlapping subjects is determined.

  As described above, when a multi-view depth signal including a plurality of viewpoint depth signals is used in addition to a multi-view image signal including a plurality of viewpoint image signals, a high-accuracy virtual viewpoint image signal is generated on the decoding side. be able to. In addition, when a multi-view image signal with a close interval between viewpoints and a multi-view depth signal corresponding to each viewpoint image signal are used, a higher-accuracy virtual viewpoint image signal can be generated on the decoding side. it can.

  However, if the number of viewpoints is set too large, the bit rate increases and transmission efficiency or storage efficiency decreases. Therefore, it is necessary to appropriately determine the viewpoint to be encoded in each of the multi-view image signal and the multi-view depth signal in consideration of the transmission rate of the target application or the capacity of the storage medium.

  At this time, it is not always necessary that the viewpoints of the multi-view image signal and the depth signal to be encoded have a one-to-one correspondence. May be used. In this case, encoding can be performed more flexibly. For example, if it is necessary to encode and transmit or store all image signals actually captured, if the virtual viewpoint image signal is easy to generate, set the viewpoint of the depth signal to be encoded to be small. May be. In this case, a more efficient encoded stream can be generated. Here, the generation of the image signal of the virtual viewpoint is easy when the distance between the viewpoints of the multi-view image signal to be encoded is sufficiently close, or the difference in depth between subjects included in the content is This is the case when there is not much.

Next, an encoded stream generated by being encoded by the image encoding device 100 according to Embodiment 1 will be described.
FIG. 10 illustrates a multi-view image including an image IS from five viewpoints (viewpoint 0, viewpoint 1, viewpoint 2, viewpoint 3, and viewpoint 4) to be encoded, and three viewpoints (viewpoint 0, viewpoint 2, and viewpoint 4). It is a figure which shows the multiview depth map containing the depth DS of this. The vertical axis indicates the viewpoint direction, and the horizontal axis indicates the time direction. Further, the viewpoint 0 is set as the base viewpoint. In the MVC encoding method, the base viewpoint is a viewpoint that can be encoded or decoded without depending on other viewpoints. Only one viewpoint is set as a base viewpoint in one entire sequence including multi-viewpoint images. That is, a base viewpoint image can be encoded or decoded independently without using an image of another viewpoint as a reference image for inter-view prediction. An image of a non-base viewpoint (that is, a viewpoint other than the base viewpoint) can be encoded or decoded using an image of another viewpoint as a reference image for inter-view prediction. In the following description, a case where the multi-view image and the multi-view depth map shown in FIG. 10 are encoded will be described.

  FIG. 11 is a diagram illustrating an example in which an encoded stream generated by the image encoding device 100 according to Embodiment 1 is expressed in units of NAL units. One square block corresponds to one NAL unit. The NAL unit includes a NAL unit header which is a header part (that is, a head part) and an RBSP (Raw Byte Sequence Payload) which is raw data excluding the NAL unit header. The header part of each NAL unit always has a flag having a value of “0” (ie, “forbidden_zero_bit”) and an identifier for distinguishing whether a slice serving as an SPS, PPS or reference picture is contained (ie, “nal_ref_idc”). ") And an identifier for identifying the type of the NAL unit (ie," nal_unit_type ").

 FIG. 12 shows AVC / H. 2 is a diagram illustrating the types of NAL units defined in the H.264 encoding scheme. FIG. On the decoding side, the type of the NAL unit can be identified by referring to “nal_unit_type” that is an identifier for identifying the type of the NAL unit included in the header part of the NAL unit.

(SPS # A)
In the encoded stream shown in FIG. 11, first, an SPS # A NAL unit is generated. In SPS # A, information related to the encoding of the entire sequence of the base viewpoint image signal (in FIG. 10, the image signal of the viewpoint 0) is set. In the value of “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of SPS # A, “7” indicating SPS is set (see FIG. 12).

  FIG. 13 is a diagram showing the configuration of the SPS NAL unit. The SPS RBSP “seq_parameter_set_rbsp” includes “seq_parameter_set_data” that includes information related to coding of the entire sequence, and “rbsp_trailing_bits” that is an adjustment bit added to the end of the RBSP. “seq_parameter_set_data” includes “profile_idc” for identifying a profile. The profile here is AVC / H. 2 shows a subset of the H.264 encoding syntax.

  For example, by setting the value of “profile_idc” of SPS # A to “100”, the encoded stream becomes AVC / H. It can be shown that it conforms to the High Profile of the H.264 encoding method. In this case, a NAL unit to be described later that should refer to SPS # A is generated based on a restriction conforming to the high profile. Further, “seq_parameter_set_data” includes “seq_parameter_set_id” that is a unique number for identifying the SPS for identifying the SPS, and “seq_parameter_set_id” of SPS # A includes SPS # B and SPS # described later. An arbitrary value different from “seq_parameter_set_id” of C is set. The SPS of the image signal of the base viewpoint does not include “seq_parameter_set_mvc_extension” that includes MVC extension information related to encoding of the entire sequence described later.

(SPS # B)
Subsequently, an SPS # B NAL unit is generated. In SPS # B, information related to encoding of the entire sequence of image signals of other viewpoints excluding the base viewpoint (in FIG. 10, the signals of the images of viewpoint 1, viewpoint 2, viewpoint 3, and viewpoint 4) is set. . In the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of SPS # B, “15” indicating the subset SPS that is the SPS of the MVC extension is set.

  FIG. 14 is a diagram illustrating the configuration of the NAL unit of the subset SPS. “Subset_seq_parameter_set_rbsp”, which is an RBSP of the subset SPS, includes “seq_parameter_set_mvc_extension_rbsp” including MVC extension information related to encoding of the entire sequence in addition to “seq_parameter_set_data” including information related to encoding of the entire sequence. . The value of “profile_idc” of SPS # B includes AVC / H. “118” indicating the multiview high profile of the H.264 encoding method is set. In this specification, AVC / H. An encoding method corresponding to the multi-view high profile of the H.264 encoding method is referred to as an MVC encoding method.

  A NAL unit to be described later that should refer to SPS # B is generated based on a restriction conforming to the multi-view high profile. Furthermore, an arbitrary value different from “seq_parameter_set_id” of SPS # A and SPS # C described later is set in “seq_parameter_set_id” of SPS # B. "seq_parameter_set_mvc_extension_rbsp" specifies the number of viewpoints of the image signal to be encoded, the encoding or decoding order in the viewpoint direction, and the viewpoint to be referred to when performing inter-view prediction when encoding or decoding the image signal Information indicating the dependency relationship between the viewpoints to be included.

  In FIG. 14, “num_views_minus1” is a parameter for setting the number of viewpoints in the encoded bit string, and is a value obtained by subtracting “1” from the number of viewpoints. In the example of FIG. 10, since the multi-viewpoint image signal including the signals of the five viewpoint images of viewpoint 0, viewpoint 1, viewpoint 2, viewpoint 3, and viewpoint 4 is encoded, the value of “num_views_minus1” is “4”. Is set.

  Subsequently, “view_id [i]” has a structure in which each viewpoint is repeatedly set in succession in the encoding or decoding order in the viewpoint direction. “view_id [i]” indicates viewpoint identification information (hereinafter referred to as viewpoint ID) when the encoding or decoding order in the viewpoint direction is indicated by an index i. That is, “view_id [i]” indicates the i-th view ID in the encoding or decoding order in the view direction. Here, in this specification, it is assumed that the array index (that is, the subscript) starts from 0. For example, the top of the array “view_id [i]” is view_id [0], and the next is “view_id [1]”. Also, when expressing the order, the first is 0th and the next is 1st. That is, the viewpoint that is first encoded or decoded in the viewpoint direction is 0th, and the viewpoint that is encoded or decoded next is 1st. For example, when encoding is performed in the order of viewpoint 0, viewpoint 2, viewpoint 1, viewpoint 4, and viewpoint 3, viewpoint ID of viewpoint 0 is assigned to "view_id [0]" and viewpoint 2 is assigned to "view_id [1]". The viewpoint ID of viewpoint 1, "view_id [2]" is the viewpoint ID of viewpoint 1, "view_id [3]" is the viewpoint ID of viewpoint 4, and "view_id [4]" is the viewpoint ID of viewpoint 3 Set each.

(SPS # C)
Subsequently, an SPS # C NAL unit is generated. In SPS # C, information related to the coding of the entire sequence of depth signals for each viewpoint is set. Similar to SPS # B, the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of SPS # C is set to “15” indicating the subset SPS that is the SPS of the MVC extension. The Here, in the present embodiment, the value of “profile_idc” indicating that the multi-view depth signal is also a decodable profile is defined as “120”. Therefore, the value of “profile_idc” of SPS # C is set to “120”. Furthermore, an arbitrary value different from “seq_parameter_set_id” of SPS # A and SPS # B described above is set in “seq_parameter_set_id” of SPS # C. In “seq_parameter_set_mvc_extension_rbsp”, the number of viewpoints of the depth signal to be encoded, the encoding or decoding order in the viewpoint direction, and the viewpoint to be referred to in inter-view prediction when encoding or decoding the depth signal are specified. Dependencies between viewpoints are included.

  A parameter is set in “seq_parameter_set_mvc_extension_rbsp” in the same manner as SPS # B, which is information relating to encoding of the entire sequence of image signals of other viewpoints other than the base viewpoint. As described above, when a multi-view depth signal including image signals of three viewpoints, viewpoint 0, viewpoint 2, and viewpoint 4, is encoded in the order of viewpoint 0, viewpoint 2, and viewpoint 4, the values of the parameters are as follows: Set to First, the value of “num_views_minus1” is set to “2”, then the viewpoint ID of viewpoint 0 is set to “view_id [0]”, the viewpoint ID of viewpoint 2 is set to “view_id [1]”, and “view_id [ 2] "is set to the viewpoint ID of viewpoint 4. By making the viewpoint ID of the image signal and the depth signal common to the same viewpoint, it is possible to clearly identify the correspondence between the viewpoint of the image signal and the viewpoint of the depth signal on the decoding side.

In the present embodiment, the depth signal is encoded in the same manner as a monochrome format image. Therefore, the chroma format “chroma_format_idc” included in “seq_parameter_set_data” represents the ratio between the luminance component and the color difference component. “0” indicating monochrome is set. Up to this point, an example has been described in which the value of “profile_idc” indicating that the multi-view depth signal can be decoded is defined as “120”, but any value other than the existing “profile_idc” value has been described. But you can.
In addition, a flag indicating whether or not the depth signal sequence information is prepared in the RBSP of the NAL unit of the subset SPS, and the value of “profile_idc” of SPS # C is set to “118” indicating the multi-view high profile. You can also.

(PPS # A)
Subsequently, a PPS # A NAL unit is generated. In PPS # A, information related to coding of the entire picture of the base viewpoint image signal (in the example of FIG. 10, the image of the viewpoint 0 image) is set. In the value of “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of PPS # A, “8” indicating PPS is set (see FIG. 12).

  FIG. 15 is a diagram showing a configuration of a PPS NAL unit. The “pic_parameter_set_rbsp” that is the RBSP of the PPS includes “pic_parameter_set_id” that is a unique number for identifying the PPS for identifying the PPS. An arbitrary value different from “pic_parameter_set_id” of PPS # B and PPS # C described later is set in “pic_parameter_set_id” of PPS # A. Furthermore, “pic_parameter_set_rbsp”, which is the RBSP of the PPS, includes “seq_parameter_set_id”, which is a number for identifying the SPS to be referred to. The value of “seq_parameter_set_id” of #A is set.

(PPS # B)
Subsequently, a PPS # B NAL unit is generated. In PPS # B, information related to the encoding of the entire picture of the image signals of other viewpoints excluding the base viewpoint (here, the signals of the images of viewpoint 1 and viewpoint 2 in FIG. 10) is set. Similarly to PPS # A, “8” indicating PPS is set in the value of “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of PPS # B.

  An arbitrary value different from “pic_parameter_set_id” of PPS # A and PPS # C described later is set in “pic_parameter_set_id” of PPS # B. Furthermore, the value of “seq_parameter_set_id” of SPS # B to be referred to by PPS # B is set in “seq_parameter_set_id” of PPS # B.

(PPS # C)
Subsequently, a PPS # C NAL unit is generated. In PPS # C, the picture information of the depth signal of each viewpoint is set. Similar to PPS # A and PPS # B, “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of PPS # C is set to “8” indicating PPS. An arbitrary value different from “pic_parameter_set_id” of PPS # A and PPS # B described above is set in “pic_parameter_set_id” of PPS # C. Furthermore, the value of “seq_parameter_set_id” of SPS # C to be referred to by PPS # C is set in “seq_parameter_set_id” of PPS # C.

(Camera parameter information)
Subsequently, a NAL unit of camera parameter information # 0 is generated. This camera parameter information includes internal parameter information and external parameter information. The internal parameter information is information specific to the camera at each viewpoint, and includes coefficients such as focal length, principal point, and radial distortion (that is, lens distortion in the radial direction from the principal point) of the camera used for photographing from each viewpoint. . The external parameter information includes the arrangement information of the cameras at each viewpoint. This arrangement information can be represented by a position (x, y, z coordinate) in a three-dimensional space or a rotation angle (roll, pitch, yaw) on three axes (x, y, z axes). .

  Camera parameter information is encoded at each time. For example, camera parameter information # 0 is camera parameter information used for capturing images from slice # A00 to slice # B30, which will be described later. This camera parameter information is encoded as “Multiview acqisition information SEI” which is a type of supplementary additional information. The value of “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of camera parameter information # 0 is set to “6” indicating SEI (see FIG. 12). The camera parameter information is not a parameter directly required for decoding data encoded by VCL, but is used when generating or displaying a virtual viewpoint after decoding.

(Prefix NAL unit # A00)
Subsequently, a prefix NAL unit # A00 is generated. The prefix NAL unit is a NAL unit for encoding the viewpoint information of the slice NAL unit following the prefix NAL unit. In the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of the prefix NAL unit # A00, “14” indicating the prefix NAL unit is set (see FIG. 12).

  FIG. 16 is a diagram illustrating the configuration of the prefix NAL unit. The NAL unit header which is the header part of the prefix NAL unit includes “nal_unit_header_svc_mvc_extension” in addition to “forbidden_zero_bit” and “nal_ref_idc, nal_unit_type”. In this “nal_unit_header_svc_mvc_extension”, the viewpoint information of the slice NAL unit following the prefix NAL unit is set. In “nal_unit_header_svc_mvc_extension” of the prefix NAL unit # A00 in FIG. 11, the viewpoint information of the subsequent slice NAL unit # A00 is set.

  The prefix NAL unit “nal_unit_header_svc_mvc_extension” includes “view_id” that is a unique number for identifying the viewpoint for identifying the viewpoint of the subsequent slice NAL unit as one piece of viewpoint information. A value indicating the viewpoint 0 is set in “view_id” of the prefix NAL unit # A00. Here, a value different from the “view_id” of the other viewpoints, that is, the viewpoint 1, the viewpoint 2, the viewpoint 3, and the viewpoint 4, is defined for the “view_id” of the viewpoint 0. “View_id” of the prefix NAL unit # A00 is used as “view_id” of the slice NAL unit # A00 of the viewpoint 0 that follows. In the MVC method, “prefix_nal_unit_rbsp” which is the RBSP of the prefix NAL unit does not define any data and is empty. That is, in the MVC method, data is not set in the RBSP of the prefix NAL unit.

(Slice NAL unit # A00)
Subsequently, a slice NAL unit # A00 is generated. In slice NAL unit # A00, the image signal of viewpoint 0, which is the base viewpoint, is set in units of slices. Here, the slice of the base viewpoint is generated as a VCL NAL unit whose “nal_unit_type” indicating the type of the NAL unit is “1” or “5” (see FIG. 12). Also, the first picture in the base viewpoint image signal sequence is encoded as an IDR picture, and the subsequent pictures are encoded as non-IDR pictures.

  Since slice NAL unit # A00 is the first slice in the sequence, the value of “nal_unit_type” indicating the type of NAL unit included in the NAL unit header of slice NAL unit # A00 is the coded slice of the IDR picture. “5” indicating the presence is set (see FIG. 12). In the example of FIG. 11, one picture is encoded as one slice, but one picture may be divided into a plurality of slices and encoded.

  FIG. 17 is a diagram illustrating a configuration of a slice NAL unit whose “nal_unit_type” value is “1” or “5”. Since the “nal_unit_header_svc_mvc_extension” is not included in the NAL unit header of the slice NAL unit whose “nal_unit_type” value is “1” or “5”, viewpoint information is not set. Therefore, the viewpoint information set in “nal_unit_header_svc_mvc_extension” of the prefix NAL unit encoded before is used. That is, the viewpoint information set in “nal_unit_header_svc_mvc_extension” of prefix NAL unit # A00 is set as the viewpoint information of slice NAL unit # A00.

  Furthermore, “slice_layer_without_partitioning_rbsp”, which is the RBSP of the slice NAL unit whose “nal_unit_type” value is “1” or “5”, includes “slice_header”, “slice_data”, and “rbsp_slice_trailing_bits”. “slice_header” includes information related to coding of a slice. “slice_data” includes encoded data such as an encoding mode, a motion vector, an encoded residual signal, and the like obtained by encoding an image signal in a slice. “rbsp_slice_trailing_bits” is a bit for adjustment.

  “slice_header” includes “pic_parameter_set_id” that is a number for identifying the PPS to be referred to. In the “pic_parameter_set_id” of the slice NAL unit # A00, the value of “pic_parameter_set_id” of the PPS #A to be referred to by the slice NAL unit # A00 is set. Further, since the value of “seq_parameter_set_id” of SPS # A that should be referred to by PPS # A is set in “seq_parameter_set_id” of PPS # A, the sequence information that slice NAL unit # A00 should refer to is SPS # A. It can be clearly identified.

(Slice NAL unit # B20)
Subsequently, a slice NAL unit 1 # B20 is generated. In slice NAL unit # B20, the image signal of viewpoint 2, which is a non-basis viewpoint, is encoded in units of slices. Also, what is encoded here is a slice of the image signal of the viewpoint 2 at the same display time as the slice # A00 of the previous viewpoint 0. In the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of the slice NAL unit # B20, “20” indicating an encoded slice other than the base viewpoint is set (see FIG. 12).

  FIG. 18 is a diagram illustrating a configuration of a slice NAL unit whose “nal_unit_type” value is “20”. The NAL unit header which is the header part of the slice NAL unit whose “nal_unit_type” value is “20” includes “nal_unit_header_svc_mvc_extension” in addition to “forbidden_zero_bit” and “nal_ref_idc, nal_unit_type”. In this “nal_unit_header_svc_mvc_extension”, the viewpoint information of the slice NAL unit is set. In the “nal_unit_header_svc_mvc_extension” of the slice NAL unit whose “nal_unit_type” value is “20”, “view_id” which is a unique number for identifying the viewpoint for identifying the viewpoint of this slice NAL unit as one of the viewpoint information Is included. A value indicating the viewpoint 2 is set in “view_id” of the slice NAL unit # B20. Here, “view_id” of viewpoint 2 is set to a value different from “view_id” of viewpoints 0, 1, 3, and 4 which are other viewpoints.

  Furthermore, “slice_layer_in_scalable_extension_rbsp” that is the RBSP of the slice NAL unit whose nal_unit_type value is “20” includes “slice_header”, “slice_data”, and “rbsp_slice_trailing_bits”. “slice_header” includes information related to encoding of a slice. “slice_data” includes encoded data such as an encoding mode, a motion vector or a disparity vector, and an encoded residual signal obtained by encoding an image signal in the slice. “rbsp_slice_trailing_bits” is a bit for adjustment. “slice_header” includes “pic_parameter_set_id” that is a number for identifying the PPS to be referred to. In the “pic_parameter_set_id” of the slice NAL unit # B20, the value of the “pic_parameter_set_id” of the PPS #B to be referred to by the slice NAL unit # B20 is set. Further, since the value of “seq_parameter_set_id” of SPS # B that PPS # B should refer to is set in “seq_parameter_set_id” of PPS # B, the sequence information that slice NAL unit # B20 should refer to is SPS # B. This can be easily determined.

(Slice NAL unit # B10)
Subsequently, slice NAL units # B10, # B40, and # B30 are sequentially generated in the same manner as the slice NAL unit # B20. In slice NAL unit # B10, the image signal of viewpoint 1, which is a non-base viewpoint, is set in units of slices, and in slice NAL unit # B40, the image signal of viewpoint 4, which is a non-base viewpoint, is set in units of slices, and slices In NAL unit # B30, the image signal of viewpoint 3, which is a non-basis viewpoint, is set in units of slices.

  In addition, what is encoded here is an image signal in units of slices of viewpoint 1, viewpoint 4, and viewpoint 3 at the same display time as that of slice # A00 of viewpoint 0 and slice # B20 of viewpoint 2. Similar to the slice NAL unit # B20, the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of the slice NAL units # B10, # 40, and # 30 is an encoded slice other than the base viewpoint. “20” is set (see FIG. 12). The “view_id” of slice NAL unit # B10 has a value indicating viewpoint 1, the “view_id” of slice NAL unit # B40 has a value indicating viewpoint 4, and the “view_id” of slice NAL unit # B30 has a viewpoint. A value indicating 3 is set. Here, “view_id” of each viewpoint is set to a value different from “view_id” of other viewpoints.

  In the “pic_parameter_set_id” of the slice NAL units # B10, # B40, and # B30, the value of “pic_parameter_set_id” of the PPS #B that the slice NAL units # B10, # B40, and # B30 should refer to, respectively. In addition, since the value of “seq_parameter_set_id” of SPS # B that should be referred to by PPS # B is set in “seq_parameter_set_id” of PPS # B, slice NAL units # B10, # B40, and # B30 should be referred to It can be clearly specified that the sequence information is SPS # B.

(Slice NAL unit # C00)
Subsequently, a slice NAL unit # C00 is generated. In the slice NAL unit # C00, a depth signal corresponding to the slice NAL unit # A00 of the image signal of the viewpoint 0 is set for each slice. Here, in the present embodiment, the value of “nal_unit_type” indicating the slice NAL unit in which the depth signal is set is defined as “21”. Accordingly, “21” is set to the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of slice NAL unit # C00.

  By setting “21” in the “nal_unit_type” of the slice NAL unit in which the depth signal is set without using the existing “nal_unit_type” value, compatibility with the conventional MVC method that does not decode the depth signal is maintained. be able to. That is, when the encoded bit string is decoded by a conventional MVC decoder that does not decode a depth signal, only the image signal is decoded by ignoring the NAL unit whose “nal_unit_type” value is “21”. This is because it can be normally decrypted. Here, the value of “nal_unit_type” indicating that the depth signal is an encoded slice is defined as “21”, but “16”, “17”, “ Other values such as 18 "," 22 "or" 23 "may be used.

  Furthermore, the configuration of the slice NAL unit whose “nal_unit_type” value is “21” is defined similarly to the configuration shown in FIG. That is, the NAL unit header that is the header part of the slice NAL unit whose “nal_unit_type” value is “21” includes “nal_unit_header_svc_mvc_extension” in addition to “forbidden_zero_bit”, “nal_ref_idc”, and “nal_unit_type”.

  A value indicating the viewpoint 0 is set in “view_id” of the slice NAL unit # C00. The value of “view_id” of the slice NAL unit # C00 is equal to the value of “view_id” of the prefix NAL unit # A00 in which the viewpoint information of the slice unit # A00 corresponding to the slice NAL unit # C00 is set.

  Furthermore, “slice_layer_in_scalable_extension_rbsp” that is the RBSP of the slice NAL unit whose “nal_unit_type” value is “21” includes “slice_header”, “slice_data”, and “rbsp_slice_trailing_bits”. “slice_header” includes information related to coding of a slice. “slice_data” includes encoded data, such as an encoding mode, a motion vector or a disparity vector, and an encoded residual signal, obtained by encoding a depth signal in the slice. “rbsp_slice_trailing_bits” is a bit for adjustment.

  “slice_header” includes “pic_parameter_set_id” that is a number for identifying the PPS to be referred to. In the “pic_parameter_set_id” of the slice NAL unit # C00, the value of “pic_parameter_set_id” of the PPS #C to be referred to by the slice NAL unit # C00 is set. Further, since the value of “seq_parameter_set_id” of SPS # C that PPS # C should refer to is set in “seq_parameter_set_id” of PPS # C, the sequence information that slice NAL unit # C00 should refer to is SPS # C. It can be clearly identified.

(Slice NAL unit # C20)
Subsequently, slice NAL units # C20 and # C40 are sequentially generated in the same manner as the slice NAL unit # C00. In the slice NAL unit # C20, the depth signal of the viewpoint 2 corresponding to the image signal of the viewpoint 2 is set in slice units, and in the slice NAL unit # C40, the depth signal of the viewpoint 4 corresponding to the image signal of the viewpoint 4 is set in slice units. Set by. Similarly to the slice NAL unit # C00, “21” is set to the value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of the slice NAL units # C20 and # 40.

  A value indicating the viewpoint 2 is set in the view_id of the slice NAL unit # C20, and a value indicating the viewpoint 4 is set in the view_id of the slice NAL unit # C40. The view_id value of the slice NAL unit # C20 is equal to the view_id value of the slice unit # B20 corresponding to the slice NAL unit # C20, and the view_id value of the slice NAL unit # C40 is the slice NAL unit # C40. Is equal to the value of view_id of slice unit # B40 corresponding to.

  In the “pic_parameter_set_id” of the slice NAL units # C20 and # 40, the value of “pic_parameter_set_id” of the PPS #C to be referred to by the slice NAL units # C20 and # C40 is set. In addition, since the value of “seq_parameter_set_id” of SPS # C that PPS # C should refer to is set in “seq_parameter_set_id” of PPS # C, sequence information that slice NAL units # C20 and # 40 should refer to It can be clearly specified that it is SPS # C.

  NAL units subsequent to the NAL unit # A1 in the camera parameter information following the slice NAL unit # C40 are also generated from the camera parameter information # 0 in the same manner as the slice NAL unit # C40. In the prefix NAL unit # A01, the viewpoint information of the subsequent slice # A01 is set in the same manner as the prefix NAL unit # A00.

  In the slice NAL unit # A01, the next image signal in the encoding or decoding order of the image signal set in the slice NAL unit # A00 is set in units of slices in the same manner as in the slice NAL unit # A00. The value of “nal_unit_type” indicating the type of the NAL unit included in the NAL unit header of the slice NAL unit # A01 is set to “1” indicating that the slice is a non-IDR picture (FIG. 12). reference).

  Slice NAL units # B21, # B11, # B41, and # B31 include the following in the encoding or decoding order in the respective viewpoints of the image signals set in slice NAL units # B20, # B10, # B40, and # B30. The image signals coming in are encoded in units of slices in the same manner as the slice NAL units # B20 and # B10. In the slice NAL units # C01, # C21, and # C41, the next depth signal in the order of encoding or decoding in each viewpoint of the depth signals set in the slice NAL units # C00, # C20, and # C40 Encoding is performed in units of slices in the same manner as NAL units # C00, # C20, and # C40.

  Returning to FIG. 1 and FIG. 3, the configuration of the image coding apparatuses 100 and 100a according to Embodiment 1 will be described more specifically. Encoding management information is supplied to the encoding management unit 101 from the outside or from an encoding management information holding unit (not shown). The encoding management unit 101 newly calculates parameters as necessary.

The encoding management unit 101
(A) parameter information relating to the entire sequence of image signals (ie SPS of the image signal),
(B) Parameter information related to the entire sequence of depth signals (ie, SPS of the depth signal),
(C) Parameter information related to the picture of the image signal (ie, PPS of the image signal), (d) Parameter information related to the picture of the depth signal (ie, PPS of the depth signal), (e) Header information related to the slice (ie the slice header of the image signal),
(F) Header information related to a slice of a picture of a depth signal (that is, a slice header of a depth signal)
It manages information related to encoding, including.

  Further, the encoding management unit 101 manages the viewpoint information of the multi-view image signal and the multi-view depth signal, the reference dependency relationship of the encoding target image, and the encoding or decoding order. As the viewpoint information, the encoding management unit 101 manages the correspondence between the image signal and the depth signal at each viewpoint using the viewpoint ID.

  The encoding management unit 101 manages whether or not to refer to an image signal or a depth signal of another viewpoint for each viewpoint, as the reference dependency relationship. In addition, as the reference dependency, the encoding management unit 101 uses the image signal or depth signal of another viewpoint as a reference image when encoding the encoding target image signal or the encoding target depth signal in units of pictures or slices. Whether to perform inter-view prediction (for example, parallax compensation prediction) to be used is managed. Further, the encoding management unit 101, as the reference dependency relationship, a decoded image signal or a decoded depth signal obtained by decoding on the encoding side after the encoding target image signal or the encoding target depth signal is encoded, It is managed whether or not it is used as a reference image when encoding an encoding target image signal or an encoding target depth signal of another viewpoint. Furthermore, the encoding management unit 101 manages which reference image should be referred to from among a plurality of reference image candidates as the reference dependency relationship.

  In addition, the encoding management unit 101 sets the decoding order of the decoding target image signal according to the reference dependency on the decoding side after the decoding order of the reference image to be referred to by the image signal. Manage to be. In addition, the encoding management unit 101 outputs the image signal and the depth signal in the order suitable for outputting the image signal and the depth signal of each viewpoint at the same time after decoding as the encoding or decoding order. Manage to encode.

  The sequence information encoding unit for image signal 102 encodes parameter information related to the entire sequence (that is, SPS of the image signal of the base viewpoint) of the base viewpoint image signal managed by the encoding management unit 101, Generate a bit string. This encoded bit string corresponds to the RBSP part of SPS # A of the entire encoded bit string shown in FIG. As described above, the SPS of the base viewpoint image signal is encoded according to the syntax structure of “seq_parameter_set_rbsp” that is the RBSP shown in FIG. 13.

  Further, the sequence information encoding unit for image signal 102 receives parameter information related to the entire sequence of the image signal of the non-base viewpoint managed by the encoding management unit 101 (that is, the SPS of the image signal of the non-base viewpoint). Encode to generate a coded bit string. This encoded bit string corresponds to the RBSP part of SPS # B of the entire encoded bit string shown in FIG. As described above, the SPS for the image signal of the non-basis viewpoint is encoded according to the syntax structure of “subset_seq_parameter_set_rbsp” which is the RBSP shown in FIG. Here, SPS MVC extension information is also encoded in accordance with the syntax structure shown in FIG.

  The depth signal sequence information encoding unit 103 encodes parameter information related to the entire sequence of the depth signal managed by the encoding management unit 101 (that is, the SPS of the depth signal), and generates an encoded bit string. This encoded bit string corresponds to the RBSP portion of SPS # C of the entire encoded bit string shown in FIG. As described above, the SPS of the depth signal is encoded according to the syntax structure of “subset_seq_parameter_set_rbsp” which is the RBSP shown in FIG. Here, SPS MVC extension information is also encoded in accordance with the syntax structure shown in FIG.

  The picture signal picture information encoding unit 104 encodes information related to the picture of the image signal managed by the encoding management unit 101 (that is, the PPS of the image signal), and generates an encoded bit string. This encoded bit string corresponds to the RBSP part of PPS # A and PPS # B of the entire encoded bit string shown in FIG. As described above, the PPS of the base viewpoint image signal and the PPS of the non-base viewpoint image signal are encoded according to the syntax structure of “pic_parameter_set_rbsp” which is the RBSP shown in FIG.

  The depth signal picture information encoding unit 105 encodes information related to the picture of the depth signal managed by the encoding management unit 101 (that is, the PPS of the depth signal), and generates an encoded bit string. This encoded bit string corresponds to the RBSP part of PPS # C of the entire encoded bit string shown in FIG. As described above, the PPS of the depth signal is encoded according to the syntax structure of “pic_parameter_set_rbsp” that is the RBSP shown in FIG.

  The camera parameter information encoding unit 106 encodes the camera parameter information used for photographing each viewpoint as SEI and generates an encoded bit string. Here, the camera parameter information includes internal parameter information and external parameter information. The internal parameter information is information specific to the camera at each viewpoint, and includes coefficients such as focal length, principal point, and radial distortion (that is, lens distortion in the radial direction from the principal point) of the camera used for photographing from each viewpoint. . The external parameter information includes the arrangement information of the cameras at each viewpoint. This arrangement information can be represented by a position (x, y, z coordinate) in a three-dimensional space or a rotation angle (roll, pitch, yaw) on three axes (x, y, z axes). .

  The image signal encoding unit 107 is supplied with the image signal of each viewpoint. In the example of FIG. 10, the image signal supplied to the image signal encoding unit 107 is an image signal of viewpoint 0, viewpoint 1, viewpoint 2, viewpoint 3, and viewpoint 4. The image signal encoding unit 107 encodes information related to the slice of the image signal managed by the encoding management unit 101 (that is, the slice header of the image signal) and the supplied encoding target image signal in units of slices. To generate an encoded stream.

  This encoded stream is the RBSP part of slices # A00, # B20, # B10, # B40, # B30, # A01, # B21, # B11, # B41, and # B31 of the entire encoded stream shown in FIG. It corresponds to. As described above, the slice header of the base viewpoint image signal and the supplied image signal of the base viewpoint slice unit according to the syntax structure of “slice_layer_without_partitioning_rbsp”, which is the RBSP shown in FIG. Encoded. More specifically, the image signal for each slice of the base viewpoint is encoded through processing such as intra prediction encoding, inter prediction encoding, orthogonal transform, quantization, and entropy encoding.

  Further, the slice header of the non-basis viewpoint image signal and the supplied encoding target non-basis viewpoint slice unit image signal are encoded according to the syntax structure of “slice_layer_in_scalable_extension_rbsp”, which is the RBSP shown in FIG. It becomes. When encoding an image signal, inter-view prediction or motion compensated prediction may be used. In that case, an image signal locally decoded from a picture of an already encoded image signal is used as a reference image. can do.

  The depth signal encoding unit 108 is supplied with the depth signal of each viewpoint. In the example of FIG. 10, the depth signal supplied to the depth signal encoding unit 108 is a depth map signal of viewpoint 0, viewpoint 2, and viewpoint 4. The depth signal encoding unit 108 encodes information related to the slice of the depth signal managed by the encoding management unit 101 (that is, the slice header of the depth signal) and the supplied depth signal to be encoded in units of slices. To generate an encoded stream.

  This encoded bit string corresponds to the RBSP part of slices # C00, # C20, # C40, # C01, # C21, and # C41 of the entire encoded bit string shown in FIG. As described above, the slice header of the depth signal and the supplied depth signal for each slice to be encoded are encoded according to the syntax structure of “slice_layer_in_scalable_extension_rbsp”, which is the RBSP shown in FIG. When encoding a depth signal, inter-view prediction or motion compensated prediction may be used. In this case, a depth signal that is locally decoded from a picture of an already encoded depth signal is used as a reference image. can do. The encoding method of the depth signal can use the same method as that of the gray scale image signal.

The unitization unit 109
(A) an encoded bit string of sequence information of the base viewpoint image signal generated by the image signal sequence information encoding unit 102;
(B) an encoded bit string of sequence information of a non-basis viewpoint image signal generated by the image signal sequence information encoding unit 102;
(C) an encoded bit string of the sequence information of the depth signal generated by the sequence information encoding unit 103 for the depth signal,
(D) a coded bit sequence of picture information of the base viewpoint image signal generated by the picture signal picture information encoding unit 104;
(E) a coded bit sequence of picture information of a non-basis viewpoint image signal generated by the picture information picture information coding unit 104;
(F) an encoded bit sequence of the picture information of the depth signal generated by the depth signal picture information encoding unit 105;
(G) an encoded bit string of the camera parameter information generated by the camera parameter information encoding unit 106;
(H) Information related to the slice of the base viewpoint image signal generated by the image signal encoding unit 107 (that is, the slice header of the base viewpoint image signal) and the encoded bit sequence of the base viewpoint slice unit image signal ,
(I) Information related to the slice of the non-basis viewpoint image signal generated by the image signal encoding unit 107 (that is, the slice header of the non-basis viewpoint image signal) and the image signal of the slice unit of the non-basis viewpoint In the encoded bit string, and (j) information related to the slice for the depth signal (that is, the slice header of the depth signal) and the encoded bit string of the depth signal in units of slices generated by the depth signal encoding unit 108,
NAL unit headers, which are header information for handling each encoded bit string in units of NAL units, are added to form NAL units.

  Further, the unitization unit 109 multiplexes the encoded bit sequences that are converted into NAL units as necessary, and generates the encoded bit sequence of the multi-view image shown in FIG. Further, when the encoded bit string is transmitted via a network, a packetizing unit (not shown) packetizes based on standards such as MPEG-2 system, MP4 file format, RTP, and the like. A transmission unit (not shown) transmits the packetized encoded bit string.

  Here, the NAL unit header shown in FIG. 13 is added to the encoded bit string of the sequence information of the base viewpoint image signal supplied from the image signal sequence information encoding unit 102. Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “7” indicating SPS. The encoded bit string to which the NAL unit header is added corresponds to the SPS # A NAL unit of the encoded bit string shown in FIG. Further, the NAL unit header shown in FIG. 14 is added to the encoded bit string of the sequence information of the image signal of the non-base viewpoint. Here, the value of “nal_unit_type” indicating the type of the NAL unit is set to “15” indicating the subset SPS that is the SPS of the MVC extension. The encoded bit string to which the NAL unit header is added corresponds to the SPS # B NAL unit of the entire encoded bit string shown in FIG.

  The NAL unit header shown in FIG. 14 is added to the encoded bit string of the depth signal sequence information supplied from the depth signal sequence information encoding unit 103. Here, the value of “nal_unit_type” indicating the type of the NAL unit is set to “15” indicating the subset SPS that is the SPS of the MVC extension. The encoded bit sequence to which the NAL unit header is added corresponds to the SPS # C NAL unit of the entire encoded bit sequence shown in FIG.

  The NAL unit header shown in FIG. 15 is added to the encoded bit sequence of the picture information of the base viewpoint image signal supplied from the picture signal picture information encoding unit 104. Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “8” indicating PPS. The encoded bit sequence to which the NAL unit header is added corresponds to the NPS unit of PPS # A in the entire encoded bit sequence shown in FIG. Further, the NAL unit header shown in FIG. 15 is also added to the coded bit string of the picture information of the non-base viewpoint image signal. Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “8” indicating PPS. The encoded bit string to which the NAL unit header is added corresponds to the NPS unit of the PPS # B in the entire encoded bit string shown in FIG.

  The NAL unit header shown in FIG. 15 is also added to the encoded bit string of the picture information of the depth signal supplied from the depth signal picture information encoding unit 105. Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “8” indicating PPS. The encoded bit sequence to which the NAL unit header is added corresponds to the PPS # C NAL unit of the entire encoded bit sequence shown in FIG.

  An NAL unit header for SEI is added to the encoded bit string of the camera parameter information supplied from the camera parameter information encoding unit 106. Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “6” indicating SEI. The encoded bit string to which the NAL unit header is added corresponds to the NAL units of the camera parameter information # 0 and # 1 in the entire encoded bit string shown in FIG.

  The encoded bit sequence including the slice header information of the encoded base viewpoint image signal and the encoded base viewpoint image signal supplied from the image signal encoding unit 107 includes the NAL unit header shown in FIG. Is added. Here, the value of “nal_unit_type” indicating the type of the NAL unit is set to “1” or “5” indicating the slice of the base viewpoint image signal. The encoded bit string to which the NAL unit header is added corresponds to the NAL units of slices # A00 and # A01 of the entire encoded bit string shown in FIG.

  A prefix NAL unit for encoding the viewpoint information of the base viewpoint image signal is set before the slice NAL unit of the base viewpoint image signal. The structure of the prefix NAL unit is as shown in FIG. 16. However, as described above, since the RBSP is not set in the MVC method, only the NAL unit header shown in FIG. 16 is set. Here, “14” indicating a prefix NAL unit is set to the value of “nal_unit_type” indicating the type of NAL unit. The encoded bit string obtained by encoding only the NAL unit header corresponds to the NAL units of the prefix NAL units # A00 and # A01 of the entire encoded bit string shown in FIG.

  Further, the NAL unit header shown in FIG. 18 is added to the encoded bit string including the slice header of the encoded non-base viewpoint image signal and the encoded non-base viewpoint slice unit image signal. Here, the value of “nal_unit_type” indicating the type of the NAL unit is set to “20” indicating that the slice is a non-basis viewpoint image signal. The encoded bit sequence to which the NAL unit header is added is the NAL of slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, # B31 of the entire encoded bit sequence shown in FIG. Corresponds to a unit.

  The NAL unit header shown in FIG. 18 is added to the encoded bit string including the slice header of the encoded depth signal and the encoded depth signal of the slice unit supplied from the depth signal encoding unit 108. . Here, the value of “nal_unit_type” indicating the type of NAL unit is set to “21” indicating that it is a slice of a depth signal. The encoded bit sequence to which the NAL unit header is added corresponds to the NAL units of slices # C00, # C10, # C20, # C01, # C11, and # C21 of the entire encoded bit sequence shown in FIG.

Next, a multi-viewpoint image encoding process procedure performed by the image encoding apparatuses 100 and 100a according to Embodiment 1 shown in FIGS.
FIG. 19 is a flowchart illustrating a multi-viewpoint image encoding process performed by the image encoding devices 100 and 100a according to Embodiment 1. First, the sequence information encoding unit for image signal 102 encodes parameter information related to encoding of the entire sequence of the base viewpoint image signal, and sequence information of the base viewpoint image signal (that is, the SPS of the base viewpoint image signal). ) Is generated (S101).

  Subsequently, the unitizing unit 109 performs NAL unitization by adding header information for handling in units of NAL units to the encoded bit string of the sequence information of the base viewpoint image signal obtained by the processing of step S101. (S102). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the sequence information encoding unit for image signal 102 encodes parameter information related to encoding of the entire sequence of the image signal of the non-base viewpoint, and sequence information (that is, non-base viewpoint) for the image signal of the non-base viewpoint. An encoded bit string of SPS of the viewpoint image signal is generated (S103).

  Subsequently, the unitization unit 109 adds NAL unit header information to the encoded bit string of the sequence information of the image signal of the non-basis viewpoint obtained by the process of step S104. (S104). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the depth signal sequence information encoding unit 103 encodes the parameter information related to the encoding of the entire depth signal sequence, and generates the encoded bit string of the depth signal sequence information (that is, the SPS of the depth signal). (S105).

  Subsequently, the unitizing unit 109 adds the header information for handling in units of NAL units to the encoded bit string of the sequence information of the depth signal obtained by the process of step S105, thereby forming a NAL unit (S106). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the picture information picture information encoding unit 104 encodes the parameter information related to the encoding of the entire picture of the base viewpoint image signal, and the picture information (that is, the base viewpoint image of the base viewpoint image signal). An encoded bit string of the PPS of the signal is generated (S107).

  Subsequently, the unitizing unit 109 performs NAL unitization by adding header information for handling in units of NAL units to the encoded bit sequence of picture information of the base viewpoint image signal obtained by the processing of step S107. (S108). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the picture information picture information encoding unit 104 encodes the parameter information related to the encoding of the entire picture of the non-basis viewpoint image signal, and the picture information (that is, the non-basis viewpoint image signal). An encoded bit string of PPS) of the image signal is generated (S109).

  Subsequently, the unitization unit 109 adds NAL unit header information to the encoded bit sequence of the picture information of the image signal of the non-basis viewpoint obtained by the process of step S109, thereby adding NAL units. (S110). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the depth signal picture information encoding unit 105 encodes the parameter information related to the encoding of the entire depth signal picture, and generates the encoded bit string of the depth signal picture information (that is, the PPS of the depth signal) ( S111).

  Subsequently, the unitizing unit 109 adds the header information for handling in units of NAL units to the encoded bit string of the picture information of the depth signal obtained by the process of step S111, thereby forming a NAL unit (S112). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the camera parameter information encoding unit 106 encodes the camera parameter information used for photographing each viewpoint as SEI, and generates an encoded bit string of the camera parameter information (S113).

  Subsequently, the unitizing unit 109 adds the header information to be handled in units of NAL units to the encoded bit string of the camera parameter information obtained by the process of step S113 to form a NAL unit (S114). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the unitization unit 109 encodes header information to be handled in units of NAL units including the viewpoint information of the subsequent NAL unit, and sets it as a prefix NAL unit (S115). As described above, this is because RBSP is not encoded in the MVC method. Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary.

  Subsequently, the image signal encoding unit 107 encodes information related to the slice of the base viewpoint image signal (that is, the slice header of the base viewpoint image signal) and the image signal in units of slices of the base viewpoint to be encoded. Then, an encoded bit sequence of the image signal in units of slices of the base viewpoint is generated (S116).

  Subsequently, the NAL unit is formed by adding header information for handling in units of NAL units to the encoded bit sequence of the image signal in units of slices of the base viewpoint obtained by the processing of the unitization unit 109 step S116 ( S117). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary. Although not shown in FIG. 19, when a picture is divided into a plurality of slices and encoded, the processing from step S116 to S117 is repeated.

  Subsequently, the image signal encoding unit 107 outputs information related to the slice of the image signal of the non-base viewpoint (that is, the slice header of the image signal of the non-base viewpoint) and the image signal in units of slices of the base viewpoint to be encoded. Encoding is performed to generate a coded bit string of the image signal for each slice of the non-basis viewpoint (S118).

  Subsequently, the unitizing unit 109 converts the non-basic viewpoint slice unit obtained by the processing of step S117 into the encoded bit sequence of the image signal by adding header information for handling in units of NAL units. (S119). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary. Although not shown in FIG. 19, when a picture is divided into a plurality of slices and encoded, the processing from steps S118 to S119 is repeated.

  Subsequently, the encoding management unit 101 determines whether the encoding processing of the image signals of all viewpoints to be encoded is completed at the display time (S120). When the encoding process of the image signal at the display time is completed (Y in S120), the process proceeds to step S121. When the encoding process is not completed (N in S120), the encoding process from step S118 to step S120 is repeated.

  Subsequently, the depth signal encoding unit 108 encodes the information related to the slice of the depth signal (that is, the slice header of the depth signal) and the depth signal of the slice unit to be encoded, and the encoded bit string of the slice of the depth signal. Generate (S121).

  Subsequently, the unitizing unit 109 adds the header information to be handled in units of NAL units to the encoded bit string of the depth signal in units of slices obtained by the processing in step S121, thereby forming a NAL unit (S122). Furthermore, the unitization unit 109 multiplexes with other NAL units as necessary. Although not shown in FIG. 19, when a picture is divided into a plurality of slices and encoded, the processing from step S121 to S122 is repeated.

  Subsequently, the encoding management unit 101 determines whether or not the encoding processing of the depth signals of all viewpoints to be encoded has been completed at the display time (S123). When the encoding process of the depth signal at the display time is completed (Y in S123), the process proceeds to step S121. When the encoding process is not completed (N in S123), the encoding process from step S121 to step S123 is repeated.

  Subsequently, the encoding management unit 101 determines whether or not encoding processing for all image signals and depth signals to be encoded has been completed (S124). When the encoding process of all the image signals and the depth signals is completed (Y in S124), this encoding process is finished. When the encoding process is not completed (N in S124), the encoding process from step S113 to step S124 is performed. repeat.

Next, a transmission processing procedure in the case of transmitting the encoded bit string of the multi-viewpoint image generated by the image encoding devices 100 and 100a according to Embodiment 1 shown in FIGS. 1 and 3 via the network will be described.
FIG. 20 is a flowchart illustrating a transmission processing procedure in the case of transmitting an encoded bit sequence of a multi-view image generated by the image encoding devices 100 and 100a according to Embodiment 1 via a network. The entire process shown in the flowchart of FIG. 20 is executed as necessary after each of steps S102, S104, S106, S108, S110, S112, S114, S115, S117, S119, and S122 in the flowchart of FIG. Is done.

  In the flowchart of FIG. 20, the packetizing unit (not shown) is the encoded bit string obtained by the processing of steps S102, S104, S106, S108, S110, S112, S114, S115, S117, S119, and S122 in the flowchart of FIG. Are packetized based on standards such as MPEG-2 system, MP4 file format, RTP, etc. (S201).

  Subsequently, the packetizing unit multiplexes with an encoded bit string such as audio as necessary (S202). Subsequently, a transmission unit (not shown) transmits the packetized encoded bit string as needed via a network or the like (S203).

  The encoded bit string encoded by the image encoding devices 100 and 100a according to Embodiment 1 is an existing single-view AVC / H. The decoding apparatus corresponding to the H.264 encoding method can also decode. In this case, only the base viewpoint image signal is obtained on the decoding side. For example, the encoded bit string shown in FIG. 11 encoded by the image encoding apparatuses 100 and 100a according to Embodiment 1 is decoded by a decoding apparatus corresponding to the high profile of the AVC / H.264 encoding method. can do.

In that case, AVC / H. It corresponds to the high profile of H.264 encoding method,
(A) NAL unit #A of SPS that is a NAL unit whose “nal_unit_type” is “7”,
(B) PAL NAL units #A, #B, #C, which are NAL units whose “nal_unit_type” is “8”.
(C) Slice NAL unit # A00 which is a NAL unit whose “nal_unit_type” is “1”, and (d) Slice NAL unit # A01 which is a NAL unit whose “nal_unit_type” is “5”.
Is decrypted.

  However, the PPS NAL units #B and #C are not actually used because the slice NAL units referring to these PPSs are not decoded. AVC / H. SPS NAL units #B and #C which are NAL units whose “nal_unit_type” is not “15” that does not correspond to the high profile of the H.264 encoding method are not decoded.

Similarly,
(A) Prefix NAL unit # A00 which is a NAL unit whose “nal_unit_type” is “14”,
(B) Slice NAL units # B10, # B20, # B11, # B21 whose Nal units are “nal_unit_type” “20”, and (c) Slice NAL units # whose Nal units “nal_unit_type” are “21” C00, # C10, # C20, # C01, # C11, # C21,
Does not decrypt.

  Furthermore, the encoded bit string encoded by the image encoding devices 100 and 100a according to Embodiment 1 can be decoded by a decoding device that supports the existing MVC encoding scheme. In that case, only a multi-view image signal is obtained on the decoding side. For example, the encoded bit string shown in FIG. 11 and encoded by the image encoding devices 100 and 100a according to Embodiment 1 is AVC / H. It is possible to perform decoding by a decoding device that supports the H.264 encoding multi-view high profile.

In that case, it corresponds to the multi-view high profile of the AVC / H.264 encoding method.
(A) NAL unit #A of SPS that is a NAL unit whose “nal_unit_type” is “7”,
(B) SPS NAL units #B, #C, which are NAL units whose “nal_unit_type” is “15”,
(C) NAL units #A, #B, #C of PPS whose “nal_unit_type” is a NAL unit of “8”,
(D) Prefix NAL unit # A00 which is a NAL unit whose “nal_unit_type” is “14”;
(E) Slice NAL unit # A00 which is a NAL unit whose “nal_unit_type” is “1”;
(F) Slice NAL unit # A01 whose Nal unit has “nal_unit_type” of “5”, and (g) Slice NAL units # B10, # B20, # B11, # whose Nal unit whose “nal_unit_type” is “20” B21,
Is decrypted.

  However, the SPS NAL unit #C and the PPS NAL unit #C are not actually used because the slice NAL unit referring to these SPS and PPS is not decoded. AVC / H. Slice NAL units # C00, # C10, # C20, # C01, # C11, and # C21 that are NAL units whose “nal_unit_type” is not “21” that does not correspond to the H.264 encoding multi-view high profile are decoded. do not do.

  As described above, according to the first embodiment, a multi-view image encoded bit string generated by encoding a multi-view image signal including image signals from a plurality of viewpoints, and depths from a plurality of viewpoints as auxiliary information. A multi-view image can be efficiently transmitted or stored by unitizing a multi-view depth signal bit sequence generated by encoding a multi-view depth signal including a signal as the same encoded stream. That is, the viewpoint of the image signal to be encoded can be greatly reduced, and the encoding efficiency and reproduction quality are improved.

  Furthermore, the data structure of the coded bit sequence is obtained by decoding only the base-view image signal with a conventional decoding device that decodes a single-view image, or the multi-view image signal with a conventional decoding device that decodes a multi-view image. A structure capable of decoding only a single-viewpoint two-dimensional image by realizing a scalable function. It is possible to maintain compatibility with the H.264 encoding method and the MVC method for only multi-viewpoint image signals.

  Furthermore, the same number of multi-view image signals and multi-view depth signals can be generated, and the number of viewpoints of the multi-view image signal and the depth signal is different. Therefore, it is possible to generate encoded bit strings that do not correspond one-to-one.

(Embodiment 2)
Next, an image decoding apparatus 300 that decodes encoded data encoded by the image encoding apparatuses 100 and 100a according to Embodiment 1 will be described.
FIG. 21 is a block diagram showing a configuration of image decoding apparatus 300 according to Embodiment 2 of the present invention. The image decoding apparatus 300 according to Embodiment 2 includes a decomposition unit 301, a decoding management unit 302, a parameter information decoding unit 320, an image signal decoding unit 307, and a depth information decoding unit (more specifically, a depth signal decoding unit 309). And a decoded image buffer 310. The parameter information decoding unit 320 includes a base viewpoint image signal sequence information decoding unit 303, a sequence information decoding unit 304 including MVC extension information, a picture information decoding unit 305, and a supplementary additional information decoding unit 306.

  The decomposing unit 301 is a depth information code in which encoded image data obtained by encoding a plurality of images from a plurality of different viewpoints and depth information indicating the depth of a specific space from at least one or more viewpoints are encoded. The encoded stream including the encoded data and the parameter information encoded data obtained by encoding the parameter information including the viewpoint information for specifying the plurality of viewpoints based on the plurality of images and the depth information is decomposed. This encoded stream includes the encoded stream generated by the image encoding devices 100 and 100a according to Embodiment 1. Note that the number of depth information encoded data included in the encoded stream may be set smaller than the number of image encoded data.

  The image signal decoding unit 307 decodes the image encoded data decomposed by the decomposition unit 301 to restore a plurality of images. When one viewpoint to be a reference among the plurality of viewpoints is set, the image signal decoding unit 307 encodes a first image code obtained by encoding an image from the viewpoint to be the reference among the plurality of images. The decoded data is decoded to restore the image, and the second image encoded data obtained by encoding an image other than the image from the viewpoint to be the reference is decoded to restore the image.

  The depth information decoding unit decodes the depth information encoded data decomposed by the decomposition unit 301 to restore the depth information. Here, the depth information encoded data may be data obtained by encoding depth information represented by a monochrome image from a certain viewpoint. In this case, the depth information decoding unit decodes the depth information encoded data and restores the monochrome image.

  The parameter information decoding unit 320 decodes the parameter information encoded data decomposed by the decomposition unit 301 and restores the parameter information. When one viewpoint to be a reference among the plurality of viewpoints is set, the parameter information decoding unit 320 encodes the first parameter information of the image from the viewpoint to be the reference among the plurality of images. The first parameter information encoded data is decoded to restore the first parameter information. Further, the parameter information decoding unit 320 decodes second parameter information encoded data obtained by encoding the second parameter information of an image other than the image from the viewpoint to be the reference among the plurality of images, The second parameter information is restored. The parameter information decoding unit 320 decodes the third parameter information encoded data obtained by encoding the third parameter information of the depth information, and restores the third parameter information.

  Note that the third parameter information may be described in a syntax structure corresponding to the syntax structure of the second parameter information. For example, the second parameter information and the third parameter information are AVC / H. It may be described in conformity with the multi-view high profile of the H.264 encoding method. Further, viewpoint identification information may be described in the second parameter information and the third parameter information, the position of the viewpoint that is the basis of the image encoded as the image encoded data, and the depth When the viewpoint positions that are the basis of the depth information encoded as the information encoded data match, common identification information may be given to these viewpoints.

  FIG. 22 is a block diagram showing a configuration of an image decoding device 300a according to a modification of the second embodiment. The image decoding device 300a according to the modification of the second embodiment has a configuration in which a virtual viewpoint image generation unit 330 is added to the image decoding device 300 illustrated in FIG.

  In this modification, the virtual viewpoint image generation unit 330 is different from the viewpoint based on the image based on the image decoded by the image signal decoding unit 307 and the depth information decoded by the depth information decoding unit. , Generate an image from another viewpoint. More specifically, the virtual viewpoint image generation unit 330 has the image decoded by the image signal decoding unit 307, the depth information decoded by the depth information decoding unit, and the camera parameter decoded by the parameter information decoding unit 320. An image from a virtual viewpoint is generated based on parameter information such as

  The virtual viewpoint image generation unit 330 can realize generation of an image from this virtual viewpoint using an existing algorithm. This virtual viewpoint is designated to the virtual viewpoint image generation unit 330 by an instruction from the application or due to a user operation. The other processing is the same as the description of the image decoding apparatus 300 according to the basic example of the second embodiment shown in FIG.

  Hereinafter, the configuration of the image decoding devices 300 and 300a according to Embodiment 2 will be described more specifically. The decomposition unit 301 acquires the encoded bit string generated by the image encoding devices 100 and 100a according to Embodiment 1. The form for obtaining the encoded bit string may be a form for receiving the encoded bit string transmitted over the network, a form for reading the encoded bit string recorded on a storage medium such as a DVD, or a broadcast such as BS / terrestrial wave. A form in which a broadcast coded bit string is received may be used.

  Also, the decomposing unit 301 separates the supplied encoded bit string into NAL unit units. At this time, a packet decomposing unit (not shown) removes packet headers such as MPEG-2 system, MP4 file format, RTP, etc. as necessary. The disassembling unit 301 decodes the NAL unit header that is the header portion of the separated NAL unit, and supplies the decoded NAL unit header information to the decoding management unit 302. Management of the information of these NAL unit headers is performed by the decoding management unit 302.

  The disassembling unit 301 has a value of “nal_unit_type” that is an identifier for identifying the type of the NAL unit included in the NAL unit header, which is “7”, that is, the NAL unit relates to encoding of the entire sequence of the base viewpoint image signal. In the case of an encoded bit string in which parameter information is encoded, the encoded bit string of the RBSP part of the NAL unit is supplied to the base-view image signal sequence information decoding unit 303.

  When the value of “nal_unit_type” is “15”, that is, in the case of an encoded bit string in which parameter information related to encoding of the entire sequence including MVC extension information is encoded, the decomposing unit 301 encodes the code of the RBSP unit of the NAL unit The generated bit string is supplied to the sequence information decoding unit 304 including the MVC extension information.

  When the value of “nal_unit_type” is “8”, that is, in the case of an encoded bit string in which parameter information related to the encoding of a picture is encoded, the decomposing unit 301 converts the encoded bit string of the RBSP part of the NAL unit into picture information It supplies to the decoding part 305.

  When the value of “nal_unit_type” is “6”, that is, the encoded bit string in which the supplementary additional information is encoded, the decomposing unit 301 supplies the encoded bit string of the RBSP unit of the NAL unit to the supplementary additional information decoding unit 306. To do.

  The decomposition unit 301 has a value of “nal_unit_type” of “1” or “5”, that is, a code in which an encoding mode, a motion vector or a disparity vector, an encoded residual signal, and the like of an image signal of a base viewpoint are encoded In the case of an encoded bit sequence, the encoded bit sequence of the RBSP unit of the NAL unit is supplied to the image signal decoding unit 307.

  The decomposition unit 301 has a value of “nal_unit_type” of “20”, that is, an encoded bit string in which an encoding mode, a motion vector or a disparity vector, an encoded residual signal, etc. of an image signal of a non-basis viewpoint are encoded. In this case, the encoded bit string of the RBSP unit of the NAL unit is supplied to the image signal decoding unit 307.

  If the value of “nal_unit_type” is “21”, that is, the coded bit sequence in which the coding mode, the motion vector or the disparity vector, the coded residual signal, or the like of the depth signal is coded, The encoded bit string of the RBSP unit of the unit is supplied to the depth signal decoding unit 309.

  In the case of a prefix NAL unit in which the value of “nal_unit_type” is “14”, that is, the viewpoint information of the subsequent slice NAL unit is encoded, the encoded bit string of the RBSP portion of the NAL unit is empty.

  When the value of “nal_unit_type” is “14”, “20”, or “21”, the decomposition unit 301 also decodes “nal_unit_header_svc_mvc_extension” that is the viewpoint information included in the NAL unit header, and decodes and manages the decoded viewpoint information Supplied to the unit 302. The viewpoint information decoded here includes a viewpoint ID and the like. The viewpoint information included in the NAL unit header whose “nal_unit_type” value is “14” is the viewpoint information of the subsequent NAL unit, and is included in the NAL unit header whose “nal_unit_type” value is “20” or “21”. The viewpoint information to be displayed is the viewpoint information of the NAL unit. The management of these viewpoint information is performed by the decoding management unit 302.

  The base viewpoint image signal sequence information decoding unit 303 decodes an encoded bit string in which parameter information relating to encoding of the entire sequence of the base viewpoint image signal supplied from the decomposition unit 301 is encoded. The supplied encoded bit string corresponds to the RBSP part of SPS # A of the encoded bit string shown in FIG. Here, the supplied encoded bit string of the RBSP part is “seq_parameter_set_rbsp” shown in FIG. The base viewpoint image signal sequence information decoding unit 303 decodes the encoded bit string according to the syntax structure of “seq_parameter_set_rbsp” shown in FIG. 13, and sets parameter information related to the encoding of the entire sequence of the base viewpoint image signal. obtain. The base viewpoint image signal sequence information decoding unit 303 supplies sequence information of the decoded base viewpoint image signal to the decoding management unit 302. The decoding management unit 302 manages sequence information of the base viewpoint image signal.

  The sequence information decoding unit 304 including the MVC extension information is parameter information related to the encoding of the entire sequence including the MVC extension information supplied from the decomposition unit 301, that is, the sequence information of the image signal of the non-basis viewpoint or the sequence information of the depth signal Decode the encoded bit string encoded by. The supplied encoded bit string corresponds to the RBSP part of SPS # B and SPS # C of the encoded bit string shown in FIG. Here, the supplied encoded bit string of the RBSP part is “subset_seq_parameter_set_rbsp” shown in FIG. The sequence information decoding unit 304 including the MVC extension information decodes the encoded bit string in accordance with the “subset_seq_parameter_set_rbsp” syntax structure shown in FIG. 14, and parameter information or depth related to the encoding of the entire sequence of the image signal of the non-basis viewpoint. Parameter information relating to the coding of the entire signal sequence is obtained.

  Whether the sequence information of the non-basis viewpoint image signal or the depth signal can be determined by referring to the value of “profile_idc”. The value of “profile_idc” is AVC / H. In the case of “118” indicating the multi-view high profile of the H.264 encoding method, it is the sequence information of the image signal of the non-basis viewpoint, and in the case of “120” indicating that the multi-view depth signal is a profile that can be decoded, This is the sequence information of the depth signal. “subset_seq_parameter_set_rbsp” includes MVC extension information, and the sequence information decoded by the sequence information decoding unit 304 including the MVC extension information also includes MVC extension information. The sequence information decoding unit 304 including the MVC extension information supplies the decoded non-basis viewpoint image signal sequence information or depth signal sequence information to the decoding management unit 302. Management of these sequence information is performed by the decoding management unit 302.

  The picture information decoding unit 305 decodes a coded bit string in which parameter information related to coding of the entire picture supplied from the decomposition unit 301 is coded. This supplied encoded bit string corresponds to the RBSP part of PPS # A, PPS # B, and PPS # C of the encoded bit string shown in FIG. Here, the supplied encoded bit string of the RBSP part is “pic_parameter_set_rbsp” shown in FIG. The picture information decoding unit 305 decodes the encoded bit string in accordance with the “pic_parameter_set_rbsp” syntax structure shown in FIG. 15 and encodes the entire picture of the base-view image signal, the non-base-view image signal, or the depth signal. The parameter information concerning is obtained. The picture information decoding unit 305 supplies the decoded picture information to the decoding management unit 302. This picture information is managed by the decoding manager 302.

  The supplementary additional information decoding unit 306 decodes the encoded bit string in which the supplementary additional information supplied from the decomposition unit 301 is encoded, and outputs the supplementary additional information. When camera parameter information is included in the supplied encoded bit string, this camera parameter information can be used when generating or displaying an image signal of a virtual viewpoint after decoding.

  The image signal decoding unit 307 decodes the encoded bit string in which the slice header of the base viewpoint image signal supplied from the decomposition unit 301 and the encoding mode, motion vector, encoded residual signal, and the like of the slice are encoded. To do. This supplied encoded bit string corresponds to the RBSP part of slices # A00 and # A01 of the encoded bit string shown in FIG. Here, the supplied encoded bit string of the RBSP part is “slice_layer_without_partitioning_rbsp” shown in FIG.

  The image signal decoding unit 307 decodes the encoded bit string according to the syntax structure of “slice_layer_without_partitioning_rbsp” illustrated in FIG. First, the image signal decoding unit 307 decodes “slice_header” included in “slice_layer_without_partitioning_rbsp” to obtain information related to the slice. The image signal decoding unit 307 supplies information related to the decoded slice to the decoding management unit 302.

  As described above, “slice_header” included in “slice_layer_without_partitioning_rbsp” includes the number “pic_parameter_set_id” for identifying the PPS to be referred to, and “pic_parameter_set_id” of slices # A00 and # A01 shown in FIG. Is set with the value of “pic_parameter_set_id” of PPS # A to be referred to by slices # A00 and # A01. In addition, since the value of “seq_parameter_set_id” of SPS # A to be referred to by PPS # A is set in “seq_parameter_set_id” of PPS # A, the sequence information to be referred to by slices # A00 and # A01 is SPS #. A can be clearly identified. These managements are performed by the decryption management unit 302.

  The image signal decoding unit 307 should refer to the slices # A00 and # A01 supplied from the decoding management unit 302 in addition to the information related to the slice decoded from the “slice_header” of the slice # A00 or # A01. Using the sequence information decoded from #A and the picture information decoded from PPS # A, “slice_data” included in “slice_layer_without_partitioning_rbsp” is decoded to obtain a base-view decoded image signal.

  The decoded image signal of the base viewpoint is stored in the decoded image buffer 310. When decoding the encoded bit string of the base viewpoint image signal, inter prediction such as motion compensation prediction may be used. In this case, the base viewpoint decoded image signal that has already been decoded and stored in the decoded image buffer 310 is used. Is used as a reference image. Although the viewpoint information is not included in the NAL unit header of the base viewpoint slice NAL unit, the viewpoint information of the NAL unit header of the prefix NAL unit encoded before the base viewpoint slice NAL unit is changed to the base viewpoint slice information. The viewpoint information of the slice NAL unit is used.

  Furthermore, the image signal decoding unit 307 encodes the slice header, the slice encoding mode, the motion vector or the disparity vector, the encoded residual signal, and the like of the image signal of the non-basis viewpoint supplied from the decomposition unit 301. The encoded bit string is decoded. The supplied encoded bit sequence corresponds to the RBSP portion of slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, and # B31 of the encoded bit sequence shown in FIG.

  Here, the supplied encoded bit string of the RBSP part is “slice_layer_in_scalable_extension_rbsp” shown in FIG. The image signal decoding unit 307 decodes the encoded bit string according to the syntax structure of “slice_layer_in_scalable_extension_rbsp” illustrated in FIG. First, the image signal decoding unit 307 decodes “slice_header” included in “slice_layer_in_scalable_extension_rbsp” to obtain information related to the slice. The image signal decoding unit 307 supplies information related to the decoded slice to the decoding management unit 302.

  As described above, “slice_header” included in “slice_layer_in_scalable_extension_rbsp” includes a number “pic_parameter_set_id” that identifies the PPS to be referred to, and includes slices # B20, # B10, # B40, In “pic_parameter_set_id” of # B30, # B21, # B11, # B41, and # B31, the PPS that the slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, and # B31 should refer to A value of “pic_parameter_set_id” of #B is set.

  In addition, since the value of “seq_parameter_set_id” of SPS # B to be referred to by PPS # B is set in “seq_parameter_set_id” of PPS # B, slices # B20, # B10, # B40, # B30, # B21 , # B11, # B41, # B31 can clearly specify that the sequence information to be referred to is SPS # B. These managements are performed by the decryption management unit 302.

The image signal decoding unit 307
(A) In addition to information related to slices decoded from “slice_header” of slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, and # B31,
(B) Decoded from “nal_unit_header_svc_mvc_extension” included in the NAL unit header of slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, and # B31 supplied from the decoding management unit 302 Perspective information,
(C) Sequence information decoded from SPS # B to be referenced by slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, # B31, and (d) slices # B20, # B10, # B40, # B30, # B21, # B11, # B41, # B31, picture information decoded from PPS # B to be referred to,
Is used to decode “slice_data” included in “slice_layer_in_scalable_extension_rbsp” to obtain a decoded image signal of a non-basis viewpoint.

  The decoded image signal of the non-basis viewpoint is stored in the decoded image buffer 310. Inter-prediction such as inter-view prediction and motion compensation prediction may be used when decoding the encoded bit string of the image signal of the non-basis viewpoint, but in this case, the base already decoded and stored in the decoded image buffer 310 is used. An image signal of a viewpoint or a non-basis viewpoint is used as a reference image.

  The depth signal decoding unit 309 decodes a coded bit sequence in which the slice head of the depth signal supplied from the decomposition unit 301 and the coding mode, motion vector or disparity vector of the slice, the coded residual signal, and the like are coded. . This supplied encoded bit string corresponds to the RBSP section of slices # C00, # C20, # C40, # C01, # C21, and # C41 of the encoded bit string shown in FIG.

  Here, the supplied encoded bit string of the RBSP part is “slice_layer_in_scalable_extension_rbsp” shown in FIG. The depth signal decoding unit 309 decodes the encoded bit string according to the syntax structure of “slice_layer_in_scalable_extension_rbsp” illustrated in FIG. First, the depth signal decoding unit 309 decodes “slice_header” included in “slice_layer_in_scalable_extension_rbsp”, and obtains information related to the slice. The depth signal decoding unit 309 supplies information related to the decoded slice to the decoding management unit 302.

  As described above, the “slice_header” included in “slice_layer_in_scalable_extension_rbsp” includes the number “pic_parameter_set_id” that identifies the PPS to be referred to, and includes slices # C00, # C20, # C40 illustrated in FIG. In “pic_parameter_set_id” of # C01, # C21, and # C41, the value of “pic_parameter_set_id” of PPS # C that the slices # C00, # C20, # C40, # C01, # C21, and # C41 should refer to is set. Yes. In addition, since the value of “seq_parameter_set_id” of SPS # C to be referred to by PPS # C is set in “seq_parameter_set_id” of PPS # C, slices # C00, # C20, # C40, # C01, # C21 , # C41 can clearly specify that the sequence information to be referred to is SPS # C. These managements are performed by the decryption management unit 302.

The depth signal decoding unit 309
(A) In addition to information related to slices decoded from “slice_header” of slice slices # C00, # C20, # C40, # C01, # C21, and # C41,
(B) View point information decoded from “nal_unit_header_svc_mvc_extension” included in the NAL unit header of slices # C00, # C20, # C40, # C01, # C21, and # C41 supplied from the decoding management unit 302;
(C) Sequence information decoded from SPS # C to be referenced by slices # C00, # C20, # C40, # C01, # C21, # C41, and (d) Slices # C00, # C20, # C40, # Picture information decoded from PPS # C to which C01, # C21, # C41 should refer;
Is used to decode “slice_data” included in “slice_layer_in_scalable_extension_rbsp” to obtain a decoded depth signal.

  This decoded depth signal is stored in the decoded image buffer 310. Inter-prediction such as inter-view prediction and motion compensation prediction may be used when decoding the encoded bit string of the depth signal. In this case, the decoded depth signal already decoded and stored in the decoded image buffer 310 is used. Use as a reference image. Note that the decoding method of the depth signal can use the same method as the case of the image signal in the monochrome format.

  The decoding management unit 302 manages the output timing of the decoded image signal and the decoded depth signal stored in the decoded image buffer 310, and synchronizes the decoded image signal and the decoded depth signal of each viewpoint at the same time from the decoded image buffer 310. And output. At this time, the viewpoint ID, which is information for identifying the viewpoint, is output in association with the decoded image signal and the decoded depth signal of each viewpoint.

  The decoded image signal of each viewpoint output from the image decoding devices 300 and 300a may be displayed on a display device or the like. When the desired viewpoint is not output, a virtual viewpoint image signal is generated from supplementary additional information such as a decoded image signal, a decoded depth signal, and a camera parameter output from the image decoding devices 300 and 300a, and the obtained virtual viewpoint Are displayed on a display device or the like. In the image decoding device 300a according to the modification, the virtual viewpoint image generation unit 330 may generate an image signal of the virtual viewpoint.

Next, a multi-viewpoint image decoding process procedure performed by the image decoding apparatuses 300 and 300a according to Embodiment 2 shown in FIGS.
FIG. 23 is a flowchart illustrating a multi-viewpoint image decoding process procedure performed by the image decoding apparatuses 300 and 300a according to the second embodiment. In the flowchart of FIG. 23, the decomposition unit 301 separates the acquired encoded bit string into NAL unit units, and decodes the NAL unit header (S301). The processing procedure for receiving the encoded bit string via the network and separating it in units of NAL units in step S301 will be described more specifically.

  FIG. 24 is a flowchart showing a processing procedure for receiving an encoded bit string via a network and separating it into NAL unit units. In the flowchart of FIG. 24, a receiving unit (not shown) receives an encoded bit string via the network (S401). Subsequently, a packet decomposing unit (not shown) removes the packet header added to the received encoded bit string based on the MPEG-2 system method, MP4 file format, RTP, etc. A digitized bit string is obtained (S402). Subsequently, the decomposition unit 301 separates the encoded bit string in units of NAL units (S402). Subsequently, the disassembling unit 301 decodes the NAL unit header (S403).

  Note that when the value of “nal_unit_type” is “14”, “20”, or “21”, the decomposing unit 301 also decodes “nal_unit_header_svc_mvc_extension” that is the viewpoint information included in the NAL unit header. The viewpoint information decoded here includes a viewpoint ID and the like. The viewpoint information included in the NAL unit header whose “nal_unit_type” value is “14” is the viewpoint information of the subsequent NAL unit, and is included in the NAL unit header whose “nal_unit_type” value is “20” or “21”. The viewpoint information to be displayed is the viewpoint information of the NAL unit.

Returning to the flowchart of FIG. The disassembling unit 301 evaluates “nal_unit_type” which is an identifier for identifying the type of the NAL unit included in the NAL unit header which is the header part of the NAL unit separated by the process of step S301 (S302).
(A) If “nal_unit_type” is “7”, that is, if the NAL unit is an encoded bit string obtained by encoding parameter information related to encoding of the entire sequence of the base viewpoint image signal (S302-7), step S303 Proceed to
(B) “nal_unit_type” is “15”, that is, parameter information relating to encoding of the entire sequence including MVC extension information, that is, encoding of non-basis viewpoint image signal sequence information or depth signal sequence information In the case of a bit string (15 in S302), the process proceeds to step S304.
(C) “nal_unit_type” is “8”, that is, the NAL unit is a base viewpoint image signal, a non-base viewpoint image signal, or a depth signal, which is encoded with parameter information related to the encoding of the entire picture. In the case of a bit string (S302-8), the process proceeds to step S305.
(D) When “nal_unit_type” is “6”, that is, the NAL unit is an encoded bit string in which supplementary additional information is encoded (S302-6), the process proceeds to step S306.
(E) If “nal_unit_type” is “14”, that is, if the NAL unit is a prefix NAL unit (14 in S302), the process proceeds to step S307.
(F) If “nal_unit_type” is “1” or “5”, that is, if the NAL unit is an encoded bit string obtained by encoding an image signal in the slice unit of the base viewpoint (1 or 5 in S302), the process proceeds to step S308. .
(G) If “nal_unit_type” is “20”, that is, if the NAL unit is a coded bit string obtained by coding an image signal in a slice unit of a non-basis viewpoint (step S302-20), the process proceeds to step S309.
(H) If “nal_unit_type” is “21”, that is, if the NAL unit is a coded bit string obtained by coding a depth signal in units of slices (21 in S302), the process proceeds to step S310.
(I) “nal_unit_type” may take other values (others in S302), but the description is omitted in this specification.

  The base viewpoint image signal sequence information decoding unit 303 decodes an encoded bit string in which parameter information related to encoding of the entire sequence of the base viewpoint image signal is encoded, and the base viewpoint image signal of the entire sequence. The parameter information related to the encoding of is obtained (S303).

  The sequence information decoding unit 304 including the MVC extension information is encoded by encoding the parameter information related to the encoding of the entire sequence including the MVC extension information, that is, the sequence information of the non-base viewpoint image signal or the depth signal. The bit string is decoded, and parameter information relating to the coding of the entire sequence of the image signal or depth signal of the non-basis viewpoint is obtained (S304).

  The picture information decoding unit 305 decodes an encoded bit string in which parameter information relating to encoding of the entire picture is encoded, and encodes the entire picture of a base viewpoint image signal, a non-base viewpoint image signal, or a depth signal. The parameter information concerning is obtained (S305).

  The supplementary additional information decoding unit 306 decodes the encoded bit string in which the supplementary additional information is encoded, and obtains supplementary additional information (S306).

  The decomposition unit 301 decodes the RBSP of the prefix NAL unit (S307). However, in the MVC method, since the RBSP of the prefix NAL unit is empty, the decoding process is practically not performed.

  The image signal decoding unit 307 decodes the encoded bit string in which the slice header of the base viewpoint image signal, the encoding mode of the slice of the base viewpoint image signal, the motion vector, the encoded residual signal, and the like are encoded, An image signal for each slice of the base viewpoint is obtained (S308).

  The image signal decoding unit 307 decodes a coded bit string obtained by coding a slice header of a non-basis viewpoint image signal, a coding mode of a slice of the non-basis viewpoint image signal, a motion vector, a coded residual signal, and the like. Then, an image signal for each slice of the non-base viewpoint is obtained (S309).

  The depth signal decoding unit 309 decodes the coded bit string in which the slice header of the depth signal, the coding mode of the slice of the depth signal, the motion vector, the coded residual signal, and the like are coded, and the depth signal in units of slices is decoded. Obtain (S310).

  The decoding management unit 302 determines whether it is time to output the decoded image signal and depth signal (S311). If it is not the timing to output (N in S311), the process proceeds to step S313, and if it is the timing to output (Y in S311), the decoded image signal and depth signal are output (S312), and the process proceeds to step S313. At this time, the decoded image signal and the decoded depth signal of each viewpoint and the viewpoint ID which is information for specifying the viewpoint are output in association with each other.

  It is determined whether the decoding process for all NAL units has been completed (S313). If the encoding process for all NAL units is completed (Y in S313), the decoding process is terminated. If the encoding process is not completed (N in S313), the processes from step S301 to step S313 are repeated.

  Note that the image decoding apparatuses 300 and 300a according to the second embodiment are configured such that a single-viewpoint image signal is an existing AVC / H. It is also possible to obtain a single-view image signal by decoding an encoded bit string encoded by the H.264 method. Furthermore, the image decoding apparatuses 300 and 300a according to Embodiment 2 decode a coded bit string in which a multi-view image signal that does not include a depth signal is encoded by an existing MVC method, and obtain a multi-view image signal. You can also.

  In the above description, the case where the number of viewpoints of the multi-viewpoint image and the depth map as shown in FIG. 10 is different and each does not correspond one-to-one has been described. Even if the number of multi-view depth signals is the same and each corresponds one-to-one, it can be encoded or decoded.

  As described above, according to the second embodiment, in multi-view image decoding, multi-view depth including multi-view image signals including image signals from a plurality of viewpoints and depth signals from a plurality of viewpoints as auxiliary information. A multi-view image signal and a multi-view depth signal can be obtained by decoding an encoded bit string in which the signal is encoded. At that time, the encoded bit string can be efficiently received or read out.

  In addition, the image decoding apparatuses 300 and 300a according to Embodiment 2 can decode a coded bit string in which only a conventional single-viewpoint image signal is encoded, and obtain a single-viewpoint image signal. Furthermore, the image decoding apparatuses 300 and 300a according to Embodiment 2 include an encoded bit string obtained by encoding only a multi-view image signal including an image signal of a plurality of viewpoints and not including a multi-view depth signal as auxiliary information. Multi-viewpoint image signals can also be obtained by decoding, and upward compatibility is maintained.

  Furthermore, the number of viewpoints of the multi-view image signal and the depth signal is different as well as the same number of multi-view image signals and multi-view depth signals can be decoded. It is also possible to decode encoded bit strings that do not correspond one-to-one.

(Embodiment 3)
Next, an image coding apparatus according to Embodiment 3 of the present invention will be described. The image encoding device according to Embodiment 3 determines the viewpoint of an image signal and a depth signal that need to be encoded according to the content and the contents of the scene, and the image signal and the viewpoint of the required viewpoint according to the determination. It differs from the image coding apparatus according to Embodiment 1 in that only the depth signal is coded. Since the rest is the same as that of the image coding apparatus according to Embodiment 1, the description thereof is omitted.

  FIG. 25 is a block diagram illustrating a configuration of an image encoding device 400 according to Embodiment 3. In FIG. 25, the same components as those in FIG. The image coding apparatus 400 according to Embodiment 3 has a configuration in which a determination unit 120 and switching units 121 and 122 are added to the configuration of the image coding apparatus 100 according to Embodiment 1.

  The determination unit 120 determines whether or not depth information from a certain viewpoint is to be encoded. In this case, the unitization unit 109 uses the depth signal encoding unit 108 to encode the image encoded data generated by the image signal encoding unit 107 and the depth information determined to be encoded by the determination unit 120. An encoded stream including depth information encoded data is generated.

  Further, the determination unit 120 determines whether or not an image from a certain viewpoint is to be encoded. In this case, the unitization unit 109 encodes the image determined by the determination unit 120 to be encoded by the image signal encoding unit 107 and the depth generated by the depth signal encoding unit 108. An encoded stream including information encoded data is generated. Note that the determination unit 120 can also perform both determinations. In this case, the unitization unit 109 determines that the image determined to be the encoding target by the determination unit 120 is the encoded image data encoded by the image signal encoding unit 107 and the determination unit 120 determines the encoding target. An encoded stream including depth information encoded data obtained by encoding the depth information by the depth signal encoding unit 108 is generated.

  Hereinafter, the process of the determination part 120 is demonstrated more concretely. The determination unit 120 is supplied with encoding management information, camera parameter information, an image signal for each viewpoint, and a depth signal for each viewpoint. Based on these, the determination unit 120 determines the viewpoint of the image signal to be encoded and the viewpoint of the depth signal. The determination unit 120 creates new encoding management information in which the information about the viewpoint of the image signal and the viewpoint of the depth signal that are determined not to be encoded is omitted, and supplies the encoded management information to the encoding management unit 101. Note that the encoding management information supplied to the encoding management unit 101 in FIG. 25 is the same information as the encoding management information supplied to the encoding management unit 101 in FIG.

Hereinafter, a specific example of the determination method in the determination unit 120 will be described.
As a determination example 1, the determination unit 120 has a predetermined first distance between the viewpoint that is the basis of the depth information to be determined and the viewpoint that is the source of another depth information that has already been determined as the encoding target. When it is shorter than the reference distance, it is determined that the depth information to be determined is not to be encoded, and when it is longer than the first reference distance, it is determined that the depth information to be determined is to be encoded. The first reference distance can be arbitrarily set by the designer based on knowledge obtained through experiments and simulations.

  The determination unit 120 can identify the viewpoint of each image signal and the position of the viewpoint of each depth signal from the external parameter information of the camera included in the supplied camera parameter information. The external parameter includes the arrangement information of the camera of each viewpoint, and this arrangement information includes the rotation angle on the position (x, y, z coordinate) or the three axes (x, y, z axis) in the three-dimensional space. (Roll, pitch, yaw) are included. When the interval between the viewpoints of the plurality of depth signals supplied at the same time is sufficiently close, the determination unit 120 excludes any depth signal from the encoding target. As described above, when the determination unit 120 determines that the decoding side can easily generate the image signal of the desired viewpoint even if the coding of the depth signals from some viewpoints is omitted, the image signal of the desired viewpoint is determined. The depth signal of the viewpoint that is not necessary for the generation is omitted, and the depth signal of the viewpoint that is necessary for the generation is adopted as the encoding target. This determination example 1 is based on the knowledge described with reference to FIGS.

  As a determination example 2, when the distance between the first subject and the second subject in the same image is shorter than a predetermined second reference distance, the determination unit 120 omits some of the depth signals. The second reference distance can also be arbitrarily set by the designer based on knowledge obtained through experiments and simulations. At this time, the determination unit 120 may reduce the number of depth information to be determined as an encoding target as the distance between the first subject and the second subject is shorter.

  The determination unit 120 can calculate the difference in depth between overlapping subjects from the supplied depth signal. As the depth difference between the subjects, an edge of the depth signal (for example, a point where the density changes sharply) can be extracted, and a difference in pixel values sandwiching the boundary of the edge portion can be used. When the determination unit 120 determines that the difference in depth between the overlapping subjects is sufficiently small and the decoding side can easily generate an image signal of a desired viewpoint even if coding of some viewpoints is omitted, A viewpoint depth signal that is not necessary for generating a viewpoint image signal is omitted, and a viewpoint depth signal necessary for the generation is adopted as an encoding target. This determination example 2 is based on the knowledge described with reference to FIGS.

  In the above determination examples 1 and 2, in the case of an application on the premise that the decoding side generates an image signal of a desired viewpoint, the viewpoint of the image signal can be omitted as well as the viewpoint of the depth signal.

  As a determination example 3, when the determination unit 120 predicts and generates a determination target image from another image and depth information without using the determination target image, the quality of the generated image is higher than a predetermined reference value. If it is high, it is determined that the image to be determined is not to be encoded. The reference value can also be arbitrarily set by the designer based on knowledge obtained through experiments and simulations.

  The determination unit 120 omits some of the viewpoint image signals from the supplied image signal, and predicts and generates a viewpoint image signal omitted from the remaining viewpoint image signals and depth signals. The determination unit 120 evaluates the amount of distortion between the original image signal of the omitted viewpoint and the predicted and generated image signal of the viewpoint using an index such as a square error for each pixel. The determination unit 120 determines that an image signal of a viewpoint having a distortion amount less than a predetermined reference value is a signal having a small contribution to the generation of the virtual viewpoint, and omits the image signal of the viewpoint. Although the process for omitting the image signal has been described here, the depth signal can be omitted by the same process.

  The switching unit 121 supplies only the image signal of the viewpoint to be encoded to the image signal encoding unit 107 according to the determination result of the determination unit 120. The image signal supplied to the image signal encoding unit 107 is the same signal as the image signal supplied to the image signal encoding unit 107 in FIG. Similarly, the switching unit 122 supplies only the depth signal of the viewpoint to be encoded to the depth signal encoding unit 108 according to the determination result of the determination unit 120. The image signal supplied to the depth signal encoding unit 108 is the same signal as the depth signal supplied to the depth signal encoding unit 108 of FIG.

Next, a multi-viewpoint image encoding process procedure performed by the image encoding apparatus 400 according to Embodiment 3 will be described.
FIG. 26 is a flowchart illustrating a multi-viewpoint image encoding processing procedure performed by the image encoding device 400 according to Embodiment 3. As described above, the image coding apparatus 400 according to Embodiment 3 determines the viewpoints of the image signal and the depth signal that need to be coded according to the content and the content of the scene. The image encoding processing procedure according to the third embodiment shown in FIG. 26 is that the sequence is started again when the viewpoints of the image signal and the depth signal that need to be encoded change. This is different from the image encoding processing procedure according to the first embodiment. In FIG. 26, the same steps as those in FIG. 19 are denoted by the same reference numerals, and only differences from FIG. 19 will be described.

  In the flowchart of FIG. 26, the determination unit 120 evaluates the viewpoint of the image signal to be encoded and the viewpoint of the depth signal, and determines whether to adopt the signal of the viewpoint (S501). Only the adopted signal proceeds to the processing after step S502.

  Subsequently, the encoding management unit 101 determines whether or not the viewpoints of the image signal and the depth signal adopted by the process of step S501 have changed (S502). If changed (Y in S502) and the first case, the process proceeds to step S501. If not changed (N in S502), the process proceeds to step S113.

  After step S101, the image signal and the depth signal are encoded in the same manner as the image encoding processing procedure according to the first embodiment of FIG. However, in the process of step S124, when it is determined that the encoding process for all image signals and depth signals is not completed (N of S124), the encoding process of steps S501 to S124 is repeated.

  The image encoding process and the image decoding process according to the first to third embodiments can be realized by a transmission device, a storage device, and a reception device that are equipped with hardware capable of executing the processing. It can also be realized by firmware stored in a flash memory or the like, or software such as a computer. The firmware program and software program can be provided by being recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as a data broadcast of terrestrial or satellite digital broadcasting. It is also possible.

  The present invention has been described based on some embodiments. It is understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. By the way.

1 is a block diagram showing a configuration of an image encoding device according to Embodiment 1. FIG. It is a figure which shows an example of the reference dependence relationship between images at the time of encoding the multiview image which consists of 5 viewpoints by a MVC system. 6 is a block diagram illustrating a configuration of an image encoding device according to a modification of the first embodiment. FIG. It is a figure which shows the example which image | photographs the scene where a 1st target object and a 2nd target object exist from a 2nd viewpoint and a 3rd viewpoint, and produces | generates the image of the 1st viewpoint which is a virtual viewpoint. In the example of FIG. 4, it is a figure which shows the image image | photographed, the depth map corresponding to it, and the produced | generated image. It is a figure which shows the example which image | photographs the scene where a 3rd target object and a 4th target object exist from a 5th viewpoint and a 6th viewpoint, and produces | generates the image of the 4th viewpoint which is a virtual viewpoint. In the example of FIG. 6, it is a figure which shows the image image | photographed, the depth map corresponding to it, and the produced | generated image. It is a figure which shows the example which image | photographs two scenes in which either a 5th target object or a 6th target object and a 7th target object exist from an 8th viewpoint, and produces | generates the image of the 7th viewpoint which is a virtual viewpoint. is there. FIG. 9 is a diagram illustrating a captured image, a corresponding depth map, and a generated image in the example of FIG. Includes multi-view images including images from 5 viewpoints (view 0, viewpoint 1, viewpoint 2, viewpoint 3 and viewpoint 4) to be encoded, and depth DS from 3 viewpoints (view 0, viewpoint 2 and viewpoint 4) It is a figure which shows a multiview depth map. 6 is a diagram illustrating an example in which an encoded stream generated by the image encoding device according to Embodiment 1 is expressed in units of NAL units. FIG. AVC / H. 2 is a diagram illustrating the types of NAL units defined in the H.264 encoding scheme. FIG. It is a figure which shows the structure of the NAL unit of SPS. It is a figure which shows the structure of the NAL unit of subset SPS. It is a figure which shows the structure of the NAL unit of PPS. It is a figure which shows the structure of a prefix NAL unit. It is a figure which shows the structure of the slice NAL unit whose value of "nal_unit_type" is "1" or "5". It is a figure which shows the structure of the slice NAL unit whose value of "nal_unit_type" is "20". 5 is a flowchart illustrating a multi-viewpoint image encoding process procedure by the image encoding apparatus according to Embodiment 1; 6 is a flowchart illustrating a transmission processing procedure when transmitting an encoded bit sequence of a multi-view image generated by the image encoding device according to Embodiment 1 via a network. It is a block diagram which shows the structure of the image decoding apparatus which concerns on Embodiment 2 of this invention. FIG. 10 is a block diagram illustrating a configuration of an image decoding device according to a modification of the second embodiment. 12 is a flowchart illustrating a decoding process procedure of a multi-viewpoint image by the image decoding apparatus according to Embodiment 2. It is a flowchart which shows about the process sequence which receives an encoding bit stream via a network, and isolate | separates into a NAL unit unit. 10 is a block diagram illustrating a configuration of an image encoding device according to Embodiment 3. FIG. 12 is a flowchart illustrating a multi-viewpoint image encoding processing procedure by the image encoding device according to the third embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Image encoding apparatus, 101 encoding management part, 102 Image signal sequence information encoding part, 103 Depth signal sequence information encoding part, 104 Image signal picture information encoding part, 105 Depth signal picture information encoding Unit, 106 camera parameter information encoding unit, 107 image signal encoding unit, 108 depth signal encoding unit, 109 unitization unit, 110 parameter information encoding unit, 111 depth signal generation unit, 120 determination unit, 121, 122 switching , 300, 301 decomposition unit, 302 decoding management unit, 303 base sequence image signal sequence information decoding unit, 304 sequence information decoding unit including MVC extension information, 305 picture information decoding unit, 306 supplementary additional information decoding unit, 307 Image signal decoding unit, 3 9 depth signal decoding unit 310 decoded picture buffer, 320 parameter information decoding unit, 330 a virtual viewpoint image generator.

Claims (10)

A first encoding unit that encodes a plurality of images from a plurality of different viewpoints to generate image encoded data;
A second encoding unit that encodes depth information indicating the depth of a specific space from at least one viewpoint and generates depth information encoded data;
A third encoding unit that generates parameter information encoded data by encoding parameter information including viewpoint information for specifying a plurality of viewpoints based on the plurality of images and the depth information;
Generate encoded streams including image encoded data, depth information encoded data, and parameter information encoded data generated by the first encoding unit, the second encoding unit, and the third encoding unit, respectively. A stream generator,
An image encoding device comprising: The first encoding unit generates first image encoded data by encoding an image from a viewpoint to be a reference among the plurality of images, and encodes other images to encode a second image Generate data,
The third encoding unit includes first parameter information of an image from the viewpoint to be used as the reference, second parameter information of other images, and third parameter information of the depth information among the plurality of images. Are encoded to generate first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data,
The stream generation unit is configured to generate first image encoded data, second image encoded data, and depth information encoded by the first encoding unit, the second encoding unit, and the third encoding unit, respectively. The image encoding apparatus according to claim 1, wherein an encoded stream including data, first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data is generated.   The image coding apparatus according to claim 2, wherein the third parameter information is described in a syntax structure corresponding to the syntax structure of the second parameter information. In the second parameter information and the third parameter information, viewpoint identification information is described,
The common identification information is given to the viewpoints when the position of the viewpoint that is the basis of the image and the position of the viewpoint that is the basis of the depth information match. The image encoding device described in 1. A first encoding step of encoding a plurality of images from a plurality of different viewpoints to generate encoded image data;
A second encoding step of generating depth information encoded data by encoding depth information indicating the depth of the specific space from at least one viewpoint;
A third encoding step of generating parameter information encoded data by encoding parameter information including viewpoint information for specifying a plurality of viewpoints based on the plurality of images and the depth information;
Generate encoded streams including image encoded data, depth information encoded data, and parameter information encoded data generated by the first encoding step, the second encoding step, and the third encoding step, respectively. A stream generation step;
An image encoding method comprising: The first encoding step generates a first image encoded data by encoding an image from a viewpoint to be a reference among the plurality of images, and encodes the other images to encode a second image. Generate data,
The third encoding step includes, among the plurality of images, first parameter information of an image from the viewpoint to be used as a reference, second parameter information of other images, and third parameter information of the depth information. Are encoded to generate first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data,
The stream generation step includes a first image encoded data, a second image encoded data, and a depth information encoded respectively generated by the first encoding step, the second encoding step, and the third encoding step. 6. The image encoding method according to claim 5, wherein an encoded stream including data, first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data is generated.   The image coding method according to claim 6, wherein the third parameter information is described in a syntax structure corresponding to the syntax structure of the second parameter information. A first encoding process for encoding a plurality of images from a plurality of different viewpoints and generating encoded image data;
A second encoding process for encoding depth information indicating the depth of the specific space from at least one viewpoint and generating depth information encoded data;
A third encoding process for generating parameter information encoded data by encoding parameter information including viewpoint information for specifying a plurality of viewpoints based on the plurality of images and the depth information;
Generate encoded streams including image encoded data, depth information encoded data, and parameter information encoded data generated by the first encoding process, the second encoding process, and the third encoding process, respectively. Stream generation processing,
An image encoding program executed by a computer. The first encoding process generates a first image encoded data by encoding an image from a viewpoint to be a reference among the plurality of images, and encodes other images to encode a second image. Generate data,
The third encoding process includes, among the plurality of images, first parameter information of an image from the viewpoint to be used as a reference, second parameter information of other images, and third parameter information of the depth information. Are encoded to generate first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data,
The stream generation processing includes first image encoded data, second image encoded data, and depth information encoded generated by the first encoding process, the second encoding process, and the third encoding process, respectively. 9. The image encoding program according to claim 8, wherein an encoded stream including data, first parameter information encoded data, second parameter information encoded data, and third parameter information encoded data is generated.   The image coding program according to claim 9, wherein the third parameter information is described in a syntax structure corresponding to the syntax structure of the second parameter information.