WO2012169403A1 - 画像処理装置および方法 - Google Patents
画像処理装置および方法 Download PDFInfo
- Publication number
- WO2012169403A1 WO2012169403A1 PCT/JP2012/063911 JP2012063911W WO2012169403A1 WO 2012169403 A1 WO2012169403 A1 WO 2012169403A1 JP 2012063911 W JP2012063911 W JP 2012063911W WO 2012169403 A1 WO2012169403 A1 WO 2012169403A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- index
- unit
- referred
- reference image
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
Definitions
- the present disclosure relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method capable of improving encoding efficiency in multi-viewpoint encoding.
- MPEG2 (ISO / IEC 13818-2) is defined as a general-purpose image encoding system, and is a standard that covers both interlaced scanning images and progressive scanning images, as well as standard resolution images and high-definition images.
- MPEG2 is currently widely used in a wide range of applications for professional and consumer applications.
- a code amount (bit rate) of 4 to 8 Mbps is assigned to an interlaced scanned image having a standard resolution of 720 ⁇ 480 pixels.
- a high resolution interlaced scanned image having 1920 ⁇ 1088 pixels is assigned a code amount (bit rate) of 18 to 22 Mbps.
- bit rate code amount
- MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but it did not support encoding methods with a lower code amount (bit rate) than MPEG1, that is, a higher compression rate. With the widespread use of mobile terminals, the need for such an encoding system is expected to increase in the future, and the MPEG4 encoding system has been standardized accordingly. Regarding the image coding system, the standard was approved as an international standard in December 1998 as ISO / IEC 14496-2.
- H.264 and MPEG-4 Part 10 Advanced Video Coding, hereinafter referred to as H.264 / AVC.
- HEVC High Efficiency Video (Coding) by JCTVC (Joint Collaboration (Team-Video Coding)), which is a joint standardization organization of ITU-T and ISO / IEC, for the purpose of further improving encoding efficiency than H.264 / AVC. Standardization of the encoding method is underway.
- Non-Patent Document 2 is issued as a draft.
- a reference picture index is assigned to the list L0 in descending order of POC (Picture Order Count), which is information indicating the output order of pictures. It is done.
- POC Picture Order Count
- the reference image index is assigned in the decoding order.
- the reference image index allocation method in H.264 / AVC is mainly processing within the same viewpoint. Therefore, in the case of multi-view coding, the reference image index is assigned to the reference image in the view direction after all the reference images are assigned to the reference image in the time direction. When the assigned reference image index is inefficient and it is desired to change it, a change command is sent.
- the method of assigning the reference image index in the case of multi-view coding is not efficient.
- the present disclosure has been made in view of such a situation, and can improve the encoding efficiency in multi-view encoding.
- An image processing device includes a decoding unit that decodes a bitstream to generate an image, and an index of a reference image that is referred to when predicting the image generated by the decoding unit, An index assigning unit that alternately assigns an image that can be referred to in the direction and an image that can be referred to in the time direction, and a reference image of the index assigned by the index assigning unit.
- a prediction unit for predicting the image includes a decoding unit that decodes a bitstream to generate an image, and an index of a reference image that is referred to when predicting the image generated by the decoding unit, An index assigning unit that alternately assigns an image that can be referred to in the direction and an image that can be referred to in the time direction, and a reference image of the index assigned by the index assigning unit.
- the index assigning unit can repeat the process of assigning the index of the reference image in the order of an image that can be referred to in the time direction and an image that can be referred to in the view direction.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the time direction after assigning an index of the reference image to all images that can be referred to in the view direction.
- the index assigning unit can repeat the process of assigning the index of the reference image in the order of an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the time direction after assigning an index of the reference image to all images that can be referred to in the view direction.
- the index assigning unit can refer to the time direction priority pattern assigned to an image that can be referred to in the view direction after assigning the index of the reference image to an image that can be referred to in the time direction, and the index of the reference image in the view direction. After assigning to an image, one of the view direction priority patterns to be assigned to an image that can be referred to in the time direction can be selected and an index of the reference image can be assigned.
- the index allocating unit can refer to the time direction priority pattern in which the index of the reference image is assigned in the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction, and the index of the reference image can be referred to in the view direction. It is possible to select the pattern to which the reference image is assigned from among the view direction priority patterns that repeat the process of assigning in order of images that can be referred to in the time direction from the correct image, and assign the index of the reference image.
- the index assigning unit can assign an index of the reference image according to pattern identification information for identifying whether to use the time direction priority pattern or the view direction priority pattern.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the view direction based on the view reference information of the sequence parameter set.
- the index allocation unit sets an index of the reference image based on a decoding order in the case of a P picture, and based on POC (PictureCOrder Count) in the case of a B picture. Can be assigned.
- the index assigning unit repeats the process of assigning the index of the reference image from an image that can be referred to in the time direction and assigning the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction.
- a view direction priority alternating pattern that repeats the process of assigning the index of the reference image from an image that can be referred to in the view direction and sequentially assigning the image that can be referred to in the view direction to the image that can be referred to in the time direction.
- a pattern to which a reference image index is assigned can be selected to assign the reference image index.
- the index assigning unit can assign the index of the reference image according to pattern identification information for identifying whether to use the alternating pattern with priority in the time direction or the alternating pattern with priority in the view direction.
- an image processing device generates an image by decoding a bitstream, and sets an index of a reference image that is referred to when the generated image is predicted in a view direction. Allocation is alternately performed with respect to images that can be referred to and images that can be referred to in the time direction, and the generated image is predicted with reference to the reference image of the allocated index.
- An image processing device alternately assigns an index of a reference image referred to when predicting an image to an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- a bit stream is generated by encoding using an index assigning unit, a reference image of the index assigned by the index assigning unit, and predicting the image using the image predicted by the predicting unit And an encoding unit.
- the index assigning unit can repeat the process of assigning the index of the reference image in the order of an image that can be referred to in the time direction and an image that can be referred to in the view direction.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the time direction after assigning an index of the reference image to all images that can be referred to in the view direction.
- the index assigning unit can repeat the process of assigning the index of the reference image in the order of an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the time direction after assigning an index of the reference image to all images that can be referred to in the view direction.
- the index assigning unit can refer to the time direction priority pattern assigned to an image that can be referred to in the view direction after assigning the index of the reference image to an image that can be referred to in the time direction, and the index of the reference image in the view direction.
- One of the view direction priority patterns to be assigned to an image that can be referred to in the time direction after being assigned to an image can be selected and assigned to the index of the reference image.
- the index allocating unit can refer to the time direction priority pattern in which the index of the reference image is assigned in the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction, and the index of the reference image can be referred to in the view direction. It is possible to select the pattern to which the reference image is assigned from among the view direction priority patterns that repeat the process of assigning in order of images that can be referred to in the time direction from the correct image, and assign the index of the reference image.
- It may further comprise a transmission unit that transmits pattern identification information for identifying whether to use the time direction priority pattern or the view direction priority pattern, and the bit stream generated by the encoding unit.
- the index assigning unit can assign an index of the reference image to an image that can be referred to in the view direction based on the view reference information of the sequence parameter set.
- the index allocation unit sets an index of the reference image based on a decoding order in the case of a P picture, and based on POC (PictureCOrder Count) in the case of a B picture. Can be assigned.
- the index assigning unit repeats the process of assigning the index of the reference image from an image that can be referred to in the time direction and assigning the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction.
- a view direction priority alternating pattern that repeats the process of assigning the index of the reference image from an image that can be referred to in the view direction and sequentially assigning the image that can be referred to in the view direction to the image that can be referred to in the time direction.
- a pattern to which a reference image index is assigned can be selected to assign the reference image index.
- the apparatus may further include a transmission unit that transmits pattern identification information for identifying whether to use the temporal direction-priority alternating pattern or the view direction-priority alternating pattern and the bitstream generated by the encoding unit. .
- an image processing method in which an index of a reference image that is referred to when an image processing apparatus predicts an image is an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- the reference image of the assigned index is referred to
- the image is predicted
- the image predicted by the prediction unit is encoded to generate a bitstream.
- an image is generated by decoding a bitstream.
- An index of a reference image that is referred to when predicting the generated image is alternately assigned to an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- the generated image is predicted with reference to the reference image.
- an index of a reference image that is referred to when predicting an image is alternately assigned to an image that can be referred to in the view direction and an image that can be referred to in the time direction. Then, the image is predicted with reference to the reference image of the assigned index, and is encoded using the predicted image to generate a bitstream.
- the above-described image processing apparatus may be an independent apparatus, or may be an internal block constituting one image encoding apparatus or image decoding apparatus.
- an image can be decoded.
- encoding efficiency can be improved.
- an image can be encoded.
- encoding efficiency can be improved.
- FIG. 10 is a diagram illustrating an example of a view first zig-zag (view direction priority alternate) pattern. It is a figure which shows the example of the syntax of a sequence parameter set. It is a figure which shows the example of the scene assumed to be an allocation pattern.
- FIG. 20 is a block diagram illustrating a main configuration example of a computer. It is a block diagram which shows an example of a schematic structure of a television apparatus. It is a block diagram which shows an example of a schematic structure of a mobile telephone. It is a block diagram which shows an example of a schematic structure of a recording / reproducing apparatus. It is a block diagram which shows an example of a schematic structure of an imaging device.
- FIG. 1 illustrates a configuration of an embodiment of an image encoding device as an image processing device to which the present disclosure is applied.
- the image encoding device 100 shown in FIG. 1 encodes image data using a prediction process.
- an encoding method for example, H.264. H.264 and MPEG (Moving Picture Experts Group) 4 Part 10 (AVC (Advanced Video Coding)) (hereinafter referred to as H.264 / AVC) method, HEVC (High Efficiency Video Coding) method, and the like are used.
- the image encoding device 100 includes an A / D (Analog / Digital) conversion unit 101, a screen rearrangement buffer 102, a calculation unit 103, an orthogonal transformation unit 104, a quantization unit 105, and a lossless encoding unit 106. And a storage buffer 107.
- the image encoding device 100 includes an inverse quantization unit 108, an inverse orthogonal transform unit 109, a calculation unit 110, a deblock filter 111, a decoded picture buffer 112, a selection unit 113, an intra prediction unit 114, and a motion parallax prediction / compensation unit. 115, a selection unit 116, and a rate control unit 117.
- the image encoding apparatus 100 further includes a syntax processing unit 121 and a reference image index assigning unit 122.
- the A / D conversion unit 101 performs A / D conversion on the input image data, outputs it to the screen rearrangement buffer 102, and stores it.
- the screen rearrangement buffer 102 rearranges the stored frame images in the display order in the order of frames for encoding in accordance with the GOP (Group of Picture) structure.
- the screen rearrangement buffer 102 supplies the image with the rearranged frame order to the syntax processing unit 121.
- the syntax processing unit 121 sequentially confirms the image data read from the screen rearrangement buffer 102, and inserts header information into the image data.
- the header information includes a sequence parameter set (SPS: Sequence Parameter Set) and a picture parameter set (PPS: Picture Parameter Set).
- SPS Sequence Parameter Set
- PPS Picture Parameter Set
- the syntax processing unit 121 adds a slice header (SH: Slice Header) to the head of the slice.
- the syntax processing unit 121 supplies the image in which the header information and the like are inserted to the calculation unit 103, the intra prediction unit 114, and the motion parallax prediction / compensation unit 115. In addition, the syntax processing unit 121 supplies information such as each header information and slice header to the reference image index allocation unit 122.
- the calculation unit 103 subtracts the prediction image supplied from the intra prediction unit 114 or the motion parallax prediction / compensation unit 115 via the selection unit 116 from the image supplied from the syntax processing unit 121, and orthogonalizes the difference information.
- the data is output to the conversion unit 104.
- the calculation unit 103 subtracts the prediction image supplied from the intra prediction unit 114 from the image supplied from the syntax processing unit 121.
- the arithmetic unit 103 subtracts the predicted image supplied from the motion parallax prediction / compensation unit 115 from the image supplied from the syntax processing unit 121.
- the orthogonal transform unit 104 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 103 and supplies the transform coefficient to the quantization unit 105.
- the quantization unit 105 quantizes the transform coefficient output from the orthogonal transform unit 104.
- the quantization unit 105 supplies the quantized transform coefficient to the lossless encoding unit 106.
- the lossless encoding unit 106 performs lossless encoding such as variable length encoding and arithmetic encoding on the quantized transform coefficient.
- the lossless encoding unit 106 acquires information indicating the intra prediction mode from the intra prediction unit 114, and acquires information indicating the inter prediction mode, motion vector information, and the like from the motion parallax prediction / compensation unit 115.
- the lossless encoding unit 106 encodes the quantized transform coefficient, and uses intra prediction mode information, inter prediction mode information, motion vector information, and a quantization parameter as part of the header information of the encoded data. (Multiplex).
- the lossless encoding unit 106 supplies the encoded data obtained by encoding to the accumulation buffer 107 for accumulation.
- the lossless encoding unit 106 performs lossless encoding processing such as variable length encoding or arithmetic encoding.
- lossless encoding processing such as variable length encoding or arithmetic encoding.
- variable length coding include CAVLC (Context-Adaptive Variable Length Coding).
- arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).
- the accumulation buffer 107 temporarily stores the encoded data supplied from the lossless encoding unit 106, and, for example, as a encoded image encoded at a predetermined timing, for example, a recording device or a transmission device (not shown) in the subsequent stage. Output to the road.
- the transform coefficient quantized by the quantization unit 105 is also supplied to the inverse quantization unit 108.
- the inverse quantization unit 108 inversely quantizes the quantized transform coefficient by a method corresponding to the quantization by the quantization unit 105.
- the inverse quantization unit 108 supplies the obtained transform coefficient to the inverse orthogonal transform unit 109.
- the inverse orthogonal transform unit 109 performs inverse orthogonal transform on the supplied transform coefficient by a method corresponding to the orthogonal transform processing by the orthogonal transform unit 104.
- the inversely orthogonal transformed output (restored difference information) is supplied to the calculation unit 110.
- the calculation unit 110 is supplied from the intra prediction unit 114 or the motion parallax prediction / compensation unit 115 via the selection unit 116 to the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 109, that is, the restored difference information.
- the predicted images are added to obtain a locally decoded image (decoded image).
- the calculation unit 110 adds the prediction image supplied from the intra prediction unit 114 to the difference information.
- the calculation unit 110 adds the predicted image supplied from the motion parallax prediction / compensation unit 115 to the difference information.
- the addition result is supplied to the deblock filter 111 and the decoded picture buffer 112.
- the deblocking filter 111 removes block distortion of the decoded image by appropriately performing deblocking filter processing.
- the deblocking filter 111 supplies the filter processing result to the decoded picture buffer 112.
- the decoded picture buffer 112 outputs the stored reference image to the intra prediction unit 114 or the motion parallax prediction / compensation unit 115 via the selection unit 113 at a predetermined timing.
- the decoded picture buffer 112 supplies the reference image to the intra prediction unit 114 via the selection unit 113.
- the decoded picture buffer 112 supplies the reference image to the motion parallax prediction / compensation unit 115 via the selection unit 113.
- the selection unit 113 supplies the reference image to the intra prediction unit 114 when the reference image supplied from the decoded picture buffer 112 is an image to be subjected to intra coding. In addition, when the reference image supplied from the decoded picture buffer 112 is an image to be subjected to inter coding, the selection unit 113 supplies the reference image to the motion parallax prediction / compensation unit 115.
- the intra prediction unit 114 performs intra prediction (intra-screen prediction) that generates a predicted image using the pixel values in the screen of the input image supplied from the syntax processing unit 121.
- the intra prediction unit 114 performs intra prediction in a plurality of modes (intra prediction modes).
- the intra prediction unit 114 generates predicted images in all intra prediction modes, evaluates each predicted image, and selects an optimal mode. When the optimal intra prediction mode is selected, the intra prediction unit 114 supplies the prediction image generated in the optimal mode to the calculation unit 103 and the calculation unit 110 via the selection unit 116.
- the intra prediction unit 114 supplies information such as intra prediction mode information indicating the adopted intra prediction mode to the lossless encoding unit 106 as appropriate.
- the motion parallax prediction / compensation unit 115 uses the input image supplied from the syntax processing unit 121 and the reference image supplied from the decoded picture buffer 112 via the selection unit 113 for the image to be inter-coded. Motion parallax measurement. At that time, the motion parallax prediction / compensation unit 115 uses the reference image of the reference image index allocated by the reference image index allocation unit 122. The motion parallax prediction / compensation unit 115 performs motion parallax compensation processing according to the detected motion and the parallax vector, and generates a prediction image (inter prediction image information).
- the motion parallax prediction / compensation unit 115 performs inter prediction processing in all candidate inter prediction modes, and generates a prediction image.
- the motion parallax prediction / compensation unit 115 supplies the generated predicted image to the calculation unit 103 and the calculation unit 110 via the selection unit 116.
- the motion parallax prediction / compensation unit 115 supplies the inter prediction mode information indicating the adopted inter prediction mode and the motion vector information indicating the calculated motion vector to the lossless encoding unit 106.
- the selection unit 116 supplies the output of the intra prediction unit 114 to the calculation unit 103 and the calculation unit 110 in the case of an image to be subjected to intra coding, and the output of the motion parallax prediction / compensation unit 115 in the case of an image to be subjected to inter coding. Is supplied to the calculation unit 103 and the calculation unit 110.
- the rate control unit 117 controls the quantization operation rate of the quantization unit 105 based on the compressed image stored in the storage buffer 107 so that overflow or underflow does not occur.
- the reference image index assigning unit 122 is configured to refer to the index of the reference image that is referred to when the motion parallax prediction / compensation unit 115 predicts the target image with respect to the image that can be referred to in the view direction and the image that can be referred to in the time direction. Assign them alternately.
- header information such as a sequence parameter set and a slice header from the syntax processing unit 121 is referred to as processing target image information.
- reference image time information and view information obtained from reference image header information stored in the decoded picture buffer 112 are also referred to as reference image information.
- the reference image index assigning unit 122 supplies the assigned reference image index to the motion parallax prediction / compensation unit 115.
- a reference image index can be assigned to a previously decoded image.
- the index of the reference image can be assigned to a referenceable image in the decoded picture buffer 112.
- the maximum value of the reference image index can be changed by transmitting it as header information.
- a smaller reference picture index is assigned to the list L0 in order of the decoding order of the referenceable pictures.
- FIG. 2 is a diagram showing an example of a conventional reference image index assignment method.
- PicNum represents an index of decoding order
- POC represents an index of time (picture output order)
- a square represents a picture.
- pictures arranged in POC order, pictures arranged in decoding order, and reference pictures to which a reference image index used when decoding pictures arranged in decoding order are assigned are shown.
- B2_vi0 is a B picture
- B6_vi0 is a B picture
- B1_vi0 is a B picture
- B3_vi0 is a B picture
- B5_vi0 is a B picture
- B7_vi0 is a B picture
- the above-described method of assigning the reference image index is processing within the same viewpoint.
- H. In the H.264 / AVC format and the HEVC format the reference image index is assigned to the reference image in the view direction after all the reference images are assigned to the reference image in the time direction. Then, the encoding efficiency of the assigned reference image index is low, and whenever a change is desired, a change command is sent. However, the transmission itself is one of the causes of a decrease in encoding efficiency.
- FIG. 3 is a diagram illustrating an example of a reference relationship between views in the case of a three viewpoint image.
- I picture, B2 picture, B1 picture, B2 picture, B0 picture, B2 picture, B1 picture, B2 picture, and P picture are shown in ascending order of POC (that is, ascending order of time information). Has been. Above the POC index, the PicNum index is also shown.
- View 0 is called a base view, and the image can be encoded using temporal prediction.
- View 1 and view 2 are called non-base views, and the images can be encoded using temporal prediction and disparity prediction.
- the image of the view 1 can refer to the encoded image of the view 0 and the image of the view 2 as indicated by an arrow during the parallax prediction.
- the P picture in which the POC of view 1 is the eighth is a P picture in temporal prediction, but is a B picture in disparity prediction.
- the image of the view 2 can refer to the encoded image of the view 0 as indicated by an arrow during the parallax prediction.
- the base view image is decoded. After all the other view images at the same time are decoded, the base view image at the next time (PicNum) is decoded. Decoding is performed in the order of starting.
- the reference ratio having a high reference rate is often the reference image having the closest temporal distance to the prediction target image and the images of different views at the same time as the prediction image.
- the following reference image index assignment is performed by default for the above-described three viewpoint images.
- the reference image index assigning unit 122 refers to the time information and view information of the decoded image and the time information and view information of the reference image, and assigns a reference image index to a referenceable image.
- the time information is POC, PicNum, or the like.
- the view information is a view ID and view reference information of a sequence parameter described later.
- the reference image index assignment unit 122 assigns small reference image indexes in order from the same view to different views, and assigns a small reference image index to a reference image closer to the time of the decoded image.
- the reference image index assigning unit 122 assigns the reference image index alternately to the reference image having the same view time and the reference image having the same view time.
- the reference image index may be continuously assigned to a plurality of views having the same time.
- the reference image index assignment unit 122 sets the view 0 image and the view 2 image at the same time, and all view images at the past time as referenceable images.
- the reference image index assigning unit 122 uses the view 1 and view 0 images at the same time as the reference image for the list L0, and the view at the same time for the list L1.
- the images of 1 and view 2 are set as reference images.
- the reference image index is alternately assigned to the reference image having the same view time and the reference image having the same view time.
- the order of the reference images in the view is ascending order of POC in the forward direction for the list L0, descending order of POC in the backward direction for the list L1, and in decoding order for the P picture.
- reference images having the same view time are also referred to as images that can be referred to in the time direction, and reference images having different view times are also referred to as images that can be referred to in the view direction.
- FIG. 4 will be described based on the reference image index allocation method in the case of the above three viewpoint images.
- FIG. 4 is a diagram illustrating an example of a reference image index assigning method by the reference image index assigning unit 122.
- PicNum represents an index of decoding order
- POC represents an index of time (picture output order)
- a square represents a picture.
- FIG. 4 is an example of a three-view picture of view 0 (View_id_0), view 1 (View_id_1), and view 2 (View_id_2).
- view_id_0 view 0
- view 1 View_id_1
- view 2 View_id_2
- P_vi1 is a P picture, and I_vi0, I_vi2, I_vi1, Pvi0, and P_vi2 are decoded. Note that, as described above with reference to FIG. 3, P_vi1 is a B picture between views.
- B4_vi1 is a B picture, and I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, and B4_vi2 are decoded.
- ref_idx_l1 0, which is a reference image index for the list L1
- P_vi1 which is the same view
- B2_vi1 is a B picture, and I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, and B2_vi2 are decoded.
- ref_idx_l1 0, which is the reference image index for the list L1
- B4_vi1 the same view
- B6_vi1 is a B picture, and I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, B2_vi2, B2_vi1, B6_vi0, and B6_vi2 are decoded.
- B1_vi1 is a B picture, and I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, B2_vi2, B2_vi1, B6_vi0, B6_vi2, B6_vi1, and B1_0
- ref_idx_l1 0, which is a reference image index for the list L1
- B2_vi1 the same view
- B1_vi1 is a B picture, and I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, B2_vi2, B2_vi1, B6_vi0, B6_vi2, B1_0, B1_0, B1_0, B6_vi1, B1_0 ing.
- ref_idx_l1 0, which is the reference image index for the list L1
- B4_vi1 1
- B3_vi2 which is a different view
- B5_vi1 is a B picture, I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, B2_vi2, B2_vi1, B6_vi0, B6_vi2, B6_vi1, B1_, B6_vi1, B1_ B5_vi0 and B5_vi2 are decrypted.
- ref_idx_l1 0, which is a reference image index for the list L1
- B6_vi1 1
- B5_vi2 which is a different view
- B7_vi1 is a B picture, I_vi0, I_vi2, I_vi1, Pvi0, P_vi2, P_vi1, B4_vi0, B4_vi2, B4_vi1, B2_vi0, B2_vi2, B2_vi1, B6_vi0, B6_vi2, B6_vi1, B1_, B6_vi1, 3 B5_vi0, B5_vi2, B5_vi1, B7_vi0, and B7_vi2 are decoded.
- ref_idx_l1 0, which is a reference image index for the list L1
- P_vi1 which is the same view
- the reference image index is alternately assigned to the image that can be referred to in the time direction and the image that can be referred to in the view direction.
- a small reference image index is assigned to a reference image having a high reference ratio, and as a result, the encoding efficiency can be improved.
- the smaller the reference image index the smaller the code amount. Therefore, since a small index is assigned to a reference image having a high reference ratio, the entire code amount can be reduced.
- the reference image index can be freely changed by transmitting a reference image index change command.
- the change command information needs to be encoded.
- the reference image index is often appropriately assigned, and the change command information is less transmitted. Thereby, encoding efficiency can be further improved.
- FIG. 5 is a diagram showing an example of the syntax of the sequence parameter set. The number at the left end of each line is the line number given for explanation.
- max_num_ref_frames is set in the 21st line. This max_num_ref_frames is the maximum value (number of sheets) of reference images in this stream. Thereby, the reference image index assigning unit 122 recognizes the number of reference images that can be assigned.
- View reference information is described in the 31st to 38th lines.
- the view reference information includes the total number of views, the view identifier, the number of disparity predictions in the list L0, the identifier of the reference view in the list L0, the number of disparity predictions in the list L1, and the identifier of the reference view in the list L1. Composed.
- num_views is set in the 31st line. This num_views is the total number of views contained in this stream.
- the view_id [i] is set on the 33rd line.
- This view_id [i] is an identifier for distinguishing views.
- Num_ref_views_l0 [i] is set in the 34th line. This num_ref_views_l0 [i] is the number of parallax predictions in the list L0. For example, when “num_ref_views_l0 [i]” indicates 2, it indicates that only two views can be referred to in the list L0.
- Ref_view_id_l0 [i] [j] is set in the 35th line.
- This ref_view_id_l0 [i] [j] is an identifier of a view to be referred to in the disparity prediction in the list L0. For example, even if there are three views, when “num_ref_views_l0 [i]” indicates 2, in order to identify which of the three views that the list L0 refers to is “ref_view_id_l0” [i] [j] "is set.
- Num_ref_views_l1 [i] is set in the 36th line.
- the num_ref_views_l1 [i] is the number of parallax predictions in the list L1. For example, when “num_ref_views_l1 [i]” indicates 2, it indicates that only two views can be referred to in the list L1.
- Ref_view_id_l1 [i] [j] is set in the 37th line.
- This ref_view_id_l1 [i] [j] is a view identifier to be referred to in the disparity prediction in the list L1. For example, even if there are three views, when “num_ref_views_l1 [i]” indicates 2, in order to identify which of the three views that the list L1 refers to is “ref_view_id_l1” [i] [j] "is set.
- FIG. 6 is a diagram illustrating an example of the syntax of the slice header. The number at the left end of each line is the line number given for explanation.
- slice_type is set in the fifth line. This slice_type indicates which of the I slice, P slice, and B slice this slice is.
- the view_id is set on the 8th line.
- This view_id is an ID for identifying a view. Thereby, the reference image index allocation unit 122 can identify the view.
- the pic_order_cnt_lsb is set in the 12th line.
- This pic_order_cnt_lsb is time information (that is, POC: Picture Order Count).
- Num_ref_idx_active_override_flag is set in the 14th line.
- This num_ref_idx_active_override_flag is a flag indicating whether or not to change the effective number of reference image indexes from the default. When this flag is 1, the settings on the 16th and 17th lines are valid. Although not shown, a default valid number of reference image indexes is set in the picture parameter set. When this flag is 0, the default value is valid.
- Num_ref_idx_l0_active_minus1 is set in the 16th line. This num_ref_idx_l0_active_minus1 indicates the effective number of reference image indexes in the list L0 when the flag in the 15th row is 1.
- Num_ref_idx_l1_active_minus1 is set in the 17th line. This num_ref_idx_l1_active_minus1 indicates the effective number of reference image indexes in the list L1 when the slice type in the fifth row is B slice and the flag in the 15th row is 1.
- step S101 the A / D converter 101 performs A / D conversion on the input image.
- step S102 the screen rearrangement buffer 102 stores the A / D converted image, and rearranges the picture from the display order to the encoding order.
- step S103 the syntax processing unit 121 sequentially confirms the image data read from the screen rearrangement buffer 102, and inserts header information into the image data.
- the syntax processing unit 121 supplies the image in which the header information and the like are inserted to the calculation unit 103, the intra prediction unit 114, and the motion parallax prediction / compensation unit 115.
- the syntax processing unit 121 supplies information such as a sequence parameter set and a slice header to the reference image index allocation unit 122.
- step S104 the calculation unit 103 calculates the difference between the image supplied from the syntax processing unit 121 and the predicted image.
- the predicted image is supplied from the motion parallax prediction / compensation unit 115 in the case of inter prediction or from the intra prediction unit 114 in the case of intra prediction to the calculation unit 103 via the selection unit 116.
- the data amount of difference data is reduced compared to the original image data. Therefore, the data amount can be compressed as compared with the case where the image is encoded as it is.
- step S105 the orthogonal transform unit 104 orthogonally transforms the difference information generated by the process in step S104. Specifically, orthogonal transformation such as discrete cosine transformation and Karhunen-Loeve transformation is performed, and transformation coefficients are output.
- orthogonal transformation such as discrete cosine transformation and Karhunen-Loeve transformation is performed, and transformation coefficients are output.
- step S106 the quantization unit 105 quantizes the orthogonal transform coefficient obtained by the process in step S105.
- the difference information quantized by the processing in step S106 is locally decoded as follows. That is, in step S107, the inverse quantization unit 108 inversely quantizes the quantized orthogonal transform coefficient (also referred to as a quantization coefficient) generated by the process in step S106 with characteristics corresponding to the characteristics of the quantization unit 105. To do.
- the quantized orthogonal transform coefficient also referred to as a quantization coefficient
- step S108 the inverse orthogonal transform unit 109 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by the process in step S107 with characteristics corresponding to the characteristics of the orthogonal transform unit 104.
- step S109 the calculation unit 110 adds the predicted image to the locally decoded difference information, and generates a locally decoded image (an image corresponding to an input to the calculation unit 103).
- step S110 the deblock filter 111 performs deblock filter processing on the image generated by the processing in step S109. Thereby, block distortion is removed.
- step S111 the decoded picture buffer 112 stores the image from which block distortion has been removed by the processing in step S110.
- the decoded picture buffer 112 is also supplied with an image that has not been filtered by the deblocking filter 111 from the arithmetic unit 110 and stored therein.
- step S112 the intra prediction unit 114 performs an intra prediction process in the intra prediction mode.
- step S113 the reference image index assignment unit 122 performs reference image index assignment processing. Details of the reference image index assignment processing will be described later with reference to FIG.
- the reference image index assignment unit 122 supplies the assigned reference image index to the motion parallax prediction / compensation unit 115.
- step S114 the motion parallax prediction / compensation unit 115 uses the reference image of the reference image index assigned in the process of step S113 to perform motion parallax prediction and motion parallax compensation in the inter prediction mode. I do.
- step S115 the selection unit 116 determines the optimal prediction mode based on the cost function values output from the intra prediction unit 114 and the motion parallax prediction / compensation unit 115. That is, the selection unit 116 selects either the prediction image generated by the intra prediction unit 114 or the prediction image generated by the motion parallax prediction / compensation unit 115.
- the selection information indicating which prediction image has been selected is supplied to the intra prediction unit 114 and the motion parallax prediction / compensation unit 115 for which the prediction image is selected.
- the intra prediction unit 114 supplies information indicating the optimal intra prediction mode (that is, intra prediction mode information) to the lossless encoding unit 106.
- the motion parallax prediction / compensation unit 115 When a prediction image in the optimal inter prediction mode is selected, the motion parallax prediction / compensation unit 115 performs lossless encoding unit 106 on information indicating the optimal inter prediction mode and, if necessary, information corresponding to the optimal inter prediction mode. Output to.
- Information according to the optimal inter prediction mode includes motion vector information, disparity vector information, flag information, reference frame information, and the like.
- step S116 the lossless encoding unit 106 encodes the transform coefficient quantized by the process in step S106. That is, lossless encoding such as variable length encoding or arithmetic encoding is performed on the difference image (secondary difference image in the case of inter).
- the lossless encoding unit 106 adds information regarding the prediction mode of the prediction image selected by the process of step S115 to encoded data obtained by encoding the difference image. That is, the lossless encoding unit 106 uses the intra prediction mode information supplied from the intra prediction unit 114 or the information corresponding to the optimal inter prediction mode supplied from the motion parallax prediction / compensation unit 115 as encoded data. Append.
- step S117 the accumulation buffer 107 accumulates the encoded data output from the lossless encoding unit 106.
- the encoded data stored in the storage buffer 107 is appropriately read out and transmitted to the decoding side via the transmission path.
- step S118 the rate control unit 117 controls the quantization operation rate of the quantization unit 105 so that overflow or underflow does not occur based on the compressed image accumulated in the accumulation buffer 107 by the processing in step S117. .
- step S118 ends, the encoding process ends.
- the reference image index assignment unit 122 receives the view reference information of the decoded image.
- the view reference information is described in the sequence parameter from the syntax processing unit 121 as described above with reference to FIG.
- the reference image index assignment unit 122 can recognize the number of views to be referred to in the list L0 and the list L1 and the identifier of the view by acquiring the view reference information.
- step S132 the reference image index assignment unit 122 receives the maximum value of the reference image.
- the maximum value of the reference image is set to max_num_ref_frames of the sequence parameter described above with reference to FIG.
- step S133 the reference image index assignment unit 122 determines whether the picture type of the decoded image is a P picture.
- the picture type of the decoded image is described in slice_type of the slice header described above with reference to FIG. If it is determined in step S133 that the picture type of the decoded image is a P picture, the process proceeds to step S134.
- the time information and view information of the reference image from the decoded picture buffer 112 are also referred to.
- step S134 the reference image index assigning unit 122 assigns one reference image index in the list L0 to a reference image that is the same view, has no reference image index assigned, and has the closest decoding order.
- step S135 the reference image index assigning unit 122 assigns the reference image index in the list L0 to the reference image having the closest view ID that is smaller than the view of the decoded image and is the reference image at the same time to which no reference image index is assigned. Assign one.
- step S136 the reference image index assignment unit 122 determines whether or not reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S136 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S135, and the subsequent processes are repeated.
- step S136 If it is determined in step S136 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S137.
- step S137 the reference image index assignment unit 122 determines whether the reference image index is smaller than the maximum value (number of images) of the reference image received in step S132.
- step S137 If it is determined in step S137 that the index of the reference image is smaller than the maximum value of the reference image, the process returns to step S134, and the subsequent processes are repeated. If it is determined in step S137 that the index of the reference image is greater than or equal to the maximum value of the reference image, there is no more reference image to allocate, so the reference image index allocation process is terminated, and the process proceeds to step S113 in FIG. Return.
- step S133 determines whether the picture type of the decoded image is a B picture. If it is determined in step S133 that the picture type of the decoded image is a B picture, the process proceeds to step S138.
- step S138 the reference image index assigning unit 122 assigns one reference image index in the list L0 to a reference image that is the same view, to which no reference image index is assigned, and that has the closest decoding order.
- step S139 the reference image index assigning unit 122 assigns the reference image index of the list L0 to the reference image having the closest view ID that is smaller than the view of the decoded image and is the reference image at the same time to which no reference image index is assigned. Assign one.
- step S140 the reference image index assignment unit 122 determines whether reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S140 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S139, and the subsequent processes are repeated.
- step S140 If it is determined in step S140 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S141.
- step S141 the reference image index assigning unit 122 assigns one reference image index in the list L1 to a reference image that is the same view but has no reference image index assigned and has the closest decoding order.
- step S142 the reference image index assigning unit 122 assigns the reference image index of the list L1 to the reference image having the closest view ID that is smaller than the view of the decoded image and is a reference image at the same time to which no reference image index is assigned. Assign one.
- step S143 the reference image index assignment unit 122 determines whether or not reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S143 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S142, and the subsequent processes are repeated.
- step S143 If it is determined in step S143 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S144.
- step S144 the reference image index assignment unit 122 determines whether the index of the reference image is smaller than the maximum value (number of images) of the reference image received in step S132.
- step S144 If it is determined in step S144 that the index of the reference image is smaller than the maximum value of the reference image, the process returns to step S138, and the subsequent processes are repeated. If it is determined in step S144 that the index of the reference image is equal to or greater than the maximum value of the reference image, there is no more reference image to be allocated, so the reference image index allocation process is terminated, and the process proceeds to step S113 in FIG. Return.
- the reference image index is alternately assigned to the reference image in the time direction and the reference image in the view direction.
- a small reference image index is assigned to a reference image having a high reference ratio, so that the coding efficiency can be improved.
- an index is assigned to reference images of all views at the same time and then an index is assigned to a reference image having the next closest decoding order. That is, in the above description, a case has been described in which consecutive indexes are assigned to reference images of all views at the same time, but the present invention is not limited to this.
- an index may be alternately assigned to the time direction and the view direction one by one.
- continuous indexes may be assigned only to the number of views indicated by the view reference information of the sequence parameter set in FIG.
- the number of parallax predictions in list L0 and list L1 is set in “num_ref_views_l0” and “num_ref_views_l1” in the view reference information of the sequence parameter set.
- the identifiers of views referred to in disparity prediction in “ref_view_id_l1” and “num_ref_views_l0” are also set. Therefore, when assigning an index to a reference image in the view direction, a continuous index for the view set here may be assigned based on the view reference information.
- time direction and the view direction may be determined based on the view reference information when the index is alternately assigned one by one.
- the view indicated in the view reference information is considered to have a relatively good correlation between images, and conversely, a view not indicated in the view reference information is considered to have little correlation. Therefore, coding efficiency can be further improved by using this information for index allocation, such as not assigning views not shown in view reference information to targets of allocation or preferentially assigning views shown in view reference information. Can be made.
- the index is assigned to the reference image in the time direction first, but the index is assigned to the reference image in the view direction first, and then the reference image in the time direction is assigned.
- An index may be assigned to each other. In this case as well, it is possible to assign the reference image index to all the reference images in the view direction and then assign them to the reference images in the time direction. Also at this time, the reference image in the view direction may be limited to the view indicated in the view reference information.
- the pattern in which the index is alternately assigned in the order of the reference image in the time direction and the reference image in the view direction is a time direction priority pattern.
- a pattern in which indexes are alternately assigned in the order of the reference image in the view direction and the reference image in the time direction is a view direction priority pattern.
- This pattern flag may be sent for each slice, for example.
- a better pattern differs depending on the image such as a still image or a moving image.
- the coding efficiency is improved. Can be improved.
- FIG. 9 illustrates a configuration of an embodiment of an image decoding device as an image processing device to which the present disclosure is applied.
- An image decoding apparatus 200 shown in FIG. 9 is a decoding apparatus corresponding to the image encoding apparatus 100 of FIG.
- encoded data encoded by the image encoding device 100 is transmitted to the image decoding device 200 corresponding to the image encoding device 100 via a predetermined transmission path and decoded.
- the image decoding apparatus 200 includes a storage buffer 201, a lossless decoding unit 202, an inverse quantization unit 203, an inverse orthogonal transform unit 204, a calculation unit 205, a deblock filter 206, a screen rearrangement buffer 207, And a D / A converter 208.
- the image decoding apparatus 200 includes a decoded picture buffer 209, a selection unit 210, an intra prediction unit 211, a motion parallax prediction / compensation unit 212, and a selection unit 213.
- the image decoding apparatus 200 includes a syntax processing unit 221 and a reference image index assigning unit 222.
- the accumulation buffer 201 accumulates the transmitted encoded data.
- This encoded data is encoded by the image encoding device 100.
- the syntax processing unit 221 acquires a sequence parameter set, a picture parameter set, a slice header, and the like from the encoded data read from the accumulation buffer 201 at a predetermined timing, and the acquired header information together with the encoded data. And supplied to the lossless decoding unit 202. In addition, the syntax processing unit 221 supplies the acquired header information and the like to the reference image index assignment unit 222.
- the lossless decoding unit 202 decodes the encoded data from the syntax processing unit 221 by a method corresponding to the encoding method of the lossless encoding unit 106 in FIG.
- the inverse quantization unit 203 inversely quantizes the coefficient data (quantization coefficient) obtained by decoding by the lossless decoding unit 202 by a method corresponding to the quantization method of the quantization unit 105 in FIG. That is, the inverse quantization unit 203 uses the quantization parameter supplied from the image coding apparatus 100 to perform inverse quantization of the quantization coefficient by the same method as the inverse quantization unit 108 in FIG.
- the inverse quantization unit 203 supplies the inversely quantized coefficient data, that is, the orthogonal transform coefficient, to the inverse orthogonal transform unit 204.
- the inverse orthogonal transform unit 204 is a method corresponding to the orthogonal transform method of the orthogonal transform unit 104 in FIG. Corresponding decoding residual data is obtained.
- the decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 205.
- a prediction image is supplied to the calculation unit 205 from the intra prediction unit 211 or the motion parallax prediction / compensation unit 212 via the selection unit 213.
- the calculation unit 205 adds the decoded residual data and the prediction image, and obtains decoded image data corresponding to the image data before the prediction image is subtracted by the calculation unit 103 of the image encoding device 100.
- the arithmetic unit 205 supplies the decoded image data to the deblock filter 206.
- the deblock filter 206 removes block distortion of the decoded image by appropriately performing deblock filter processing.
- the deblocking filter 206 supplies the filter processing result to the screen rearrangement buffer 207.
- the screen rearrangement buffer 207 rearranges images. That is, the order of frames rearranged for the encoding order by the screen rearrangement buffer 102 in FIG. 1 is rearranged in the original display order.
- the D / A conversion unit 208 D / A converts the image supplied from the screen rearrangement buffer 207, outputs it to a display (not shown), and displays it.
- the output of the deblock filter 206 is further supplied to the decoded picture buffer 209.
- the decoded picture buffer 209, the selection unit 210, the intra prediction unit 211, the motion parallax prediction / compensation unit 212, and the selection unit 213 are the decoded picture buffer 112, the selection unit 113, the intra prediction unit 114, the motion parallax of the image encoding device 100. This corresponds to the prediction / compensation unit 115 and the selection unit 116, respectively.
- the selection unit 210 reads the inter-processed image and the referenced image from the decoded picture buffer 209 and supplies them to the motion parallax prediction / compensation unit 212.
- the selection unit 210 reads an image used for intra prediction from the decoded picture buffer 209 and supplies the read image to the intra prediction unit 211.
- the information indicating the intra prediction mode obtained from the header information is appropriately supplied from the lossless decoding unit 202 to the intra prediction unit 211.
- the intra prediction unit 211 generates a prediction image from the reference image acquired from the decoded picture buffer 209 based on this information, and supplies the generated prediction image to the selection unit 213.
- the information (prediction mode information, motion vector information, disparity vector information, various parameters, etc.) obtained from the header information is supplied from the lossless decoding unit 202 to the motion parallax prediction / compensation unit 212. Further, the motion parallax prediction / compensation unit 212 is supplied with the reference image index allocated by the reference image index allocation unit 222.
- the motion parallax prediction / compensation unit 212 predicts from the reference image acquired from the decoded picture buffer 209 based on the information supplied from the lossless decoding unit 202 and the reference image index allocated by the reference image index allocation unit 222. Generate an image.
- the motion parallax prediction / compensation unit 212 supplies the generated predicted image to the selection unit 213.
- the selection unit 213 selects the prediction image generated by the motion parallax prediction / compensation unit 212 or the intra prediction unit 211 and supplies the selected prediction image to the calculation unit 205.
- the reference image index assigning unit 222 is basically configured in the same manner as the reference image index assigning unit 122 of the image encoding device 100.
- the reference image index assigning unit 222 uses the index of the reference image to which the motion parallax prediction / compensation unit 212 predicts the target image for the image that can be referred to in the view direction and the image that can be referred to in the time direction. Assign them alternately.
- header information such as a sequence parameter set and a slice header from the syntax processing unit 221 is referred to as target image information.
- reference image time information and view information obtained from reference image header information stored in the decoded picture buffer 209 are also referred to as reference image information.
- the reference image index assigning unit 222 supplies the assigned reference image index to the motion parallax prediction / compensation unit 212.
- step S201 the accumulation buffer 201 accumulates the transmitted encoded data.
- step S202 the syntax processing unit 221 acquires header information from the encoded data read from the accumulation buffer 201 at a predetermined timing.
- the syntax processing unit 221 supplies the acquired header information to the lossless decoding unit 202 together with the encoded data.
- the syntax processing unit 221 supplies the acquired header information and the like to the reference image index assignment unit 222.
- step S203 the lossless decoding unit 202 decodes the encoded data supplied from the syntax processing unit 221. That is, the I picture, P picture, and B picture encoded by the lossless encoding unit 106 in FIG. 1 are decoded.
- motion vector information disparity vector information
- reference frame information reference frame information
- prediction mode information intra prediction mode or inter prediction mode
- information such as a flag and a quantization parameter
- the prediction mode information is intra prediction mode information
- the prediction mode information is supplied to the intra prediction unit 211.
- the prediction mode information is inter prediction mode information
- motion vector information corresponding to the prediction mode information is supplied to the motion parallax prediction / compensation unit 212.
- step S204 the inverse quantization unit 203 performs inverse quantization on the quantized orthogonal transform coefficient obtained by decoding by the lossless decoding unit 202.
- step S205 the inverse orthogonal transform unit 204 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by inverse quantization by the inverse quantization unit 203 by a method corresponding to the orthogonal transform unit 104 in FIG. As a result, the difference information corresponding to the input of the orthogonal transform unit 104 (output of the calculation unit 103) in FIG. 1 is decoded.
- step S206 the calculation unit 205 adds the predicted image to the difference information obtained by the process in step S205. As a result, the original image data is decoded.
- step S207 the deblock filter 206 appropriately filters the decoded image obtained by the process in step S206. Thereby, block distortion is appropriately removed from the decoded image.
- step S208 the decoded picture buffer 209 stores the filtered decoded image.
- step S209 the reference image index assigning unit 222 performs reference image index assignment processing. Details of the reference image index assignment processing will be described later with reference to FIG.
- the reference image index assigning unit 222 supplies the assigned reference image index to the motion parallax prediction / compensation unit 212.
- step S210 the intra prediction unit 211 or the motion parallax prediction / compensation unit 212 performs image prediction processing corresponding to the prediction mode information supplied from the lossless decoding unit 202, respectively.
- the intra prediction unit 211 performs an intra prediction process in the intra prediction mode.
- the motion parallax prediction / compensation unit 212 uses the reference image of the reference image index allocated in the process of step S209 to perform motion parallax in the inter prediction mode. Perform prediction compensation processing.
- step S211 the selection unit 213 selects a predicted image. That is, the prediction unit 213 is supplied with the prediction image generated by the intra prediction unit 211 or the prediction image generated by the motion parallax prediction / compensation unit 212. The selection unit 213 selects the side to which the predicted image is supplied, and supplies the predicted image to the calculation unit 205. This predicted image is added to the difference information by the process of step S206.
- step S212 the screen rearrangement buffer 207 rearranges the frames of the decoded image data. That is, the order of frames of the decoded image data rearranged for encoding by the screen rearrangement buffer 102 (FIG. 1) of the image encoding device 100 is rearranged to the original display order.
- step S213 the D / A conversion unit 208 performs D / A conversion on the decoded image data in which the frames are rearranged in the screen rearrangement buffer 207.
- the decoded image data is output to a display (not shown), and the image is displayed.
- Reference image index assignment process flow Next, an example of the flow of reference image index assignment processing executed in step S209 of FIG. 10 will be described with reference to the flowchart of FIG. This process is executed only when the decoded image is a P picture or a B picture. Further, since the reference image index assignment process in FIG. 11 is basically the same process as the reference image index assignment process in FIG. 8, the effects on the process in FIG. This also applies to the processing of FIG.
- information such as a sequence parameter set of the decoded image and a slice header is supplied from the syntax processing unit 221 to the reference image index assigning unit 222.
- the reference image index assignment unit 222 receives view reference information.
- the view reference information is described in the sequence parameter from the syntax processing unit 221 as described above with reference to FIG.
- the reference image index assignment unit 222 can recognize the number of views to be referred to in the list L0 and the list L1 and the view identifier by acquiring the view reference information.
- step S232 the reference image index assignment unit 222 receives the maximum value of the reference image.
- the maximum value of the reference image is set to max_num_ref_frames of the sequence parameter described above with reference to FIG.
- step S233 the reference image index assignment unit 222 determines whether or not the picture type of the decoded image is a P picture.
- the picture type of the decoded image is described in slice_type of the slice header described above with reference to FIG.
- step S233 the process proceeds to step S234.
- the time information and view information of the reference image from the decoded picture buffer 209 are also referred to.
- step S234 the reference image index assigning unit 222 assigns one reference image index in the list L0 to a reference image that is not assigned a reference image index and has the closest decoding order in the same view.
- step S235 the reference image index assigning unit 222 assigns the reference image index of the list L0 to the reference image having the closest view ID that is smaller than the view of the decoded image and is the reference image at the same time to which no reference image index is assigned. Assign one.
- step S236 the reference image index assignment unit 222 determines whether or not reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S236 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S235, and the subsequent processes are repeated.
- step S236 If it is determined in step S236 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S237.
- step S237 the reference image index assignment unit 222 determines whether or not the reference image index is smaller than the maximum value (number of images) of the reference image received in step S232.
- step S237 If it is determined in step S237 that the index of the reference image is smaller than the maximum value of the reference image, the process returns to step S234, and the subsequent processes are repeated. If it is determined in step S237 that the index of the reference image is equal to or greater than the maximum value of the reference image, there is no more reference image to be allocated, so the reference image index allocation process is terminated, and the process proceeds to step S209 in FIG. Return.
- step S233 if it is determined in step S233 that the picture type of the decoded image is a B picture, the process proceeds to step S238.
- step S2308 the reference image index assigning unit 222 assigns one reference image index in the list L0 to a reference image that is the same view but has no reference image index assigned and has the closest decoding order.
- step S239 the reference image index assigning unit 222 assigns the reference image index of the list L0 to the reference image having the closest view ID that is smaller than the view of the decoded image and is the reference image at the same time to which no reference image index is assigned. Assign one.
- step S240 the reference image index assignment unit 222 determines whether or not reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S240 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S239, and the subsequent processes are repeated.
- step S240 If it is determined in step S240 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S241.
- step S241 the reference image index assigning unit 222 assigns one reference image index in the list L1 to a reference image that is the same view but has no reference image index assigned and has the closest decoding order.
- the reference image index assigning unit 222 assigns the reference image index of the list L1 to the reference image having the closest view ID that is smaller than the view of the decoded image and is the reference image at the same time to which no reference image index is assigned. Assign one.
- step S243 the reference image index assignment unit 222 determines whether or not reference image index assignment has been completed for the reference images of all views at the same time. If it is determined in step S243 that the reference image index assignment has not been completed for the reference images of all views at the same time, the process returns to step S242, and the subsequent processes are repeated.
- step S243 If it is determined in step S243 that the reference image index assignment has been completed for the reference images of all views at the same time, the process proceeds to step S244.
- step S244 the reference image index assignment unit 222 determines whether the reference image index is smaller than the maximum value (number of sheets) of the reference image received in step S232.
- step S244 If it is determined in step S244 that the index of the reference image is smaller than the maximum value of the reference image, the process returns to step S238, and the subsequent processes are repeated. If it is determined in step S244 that the index of the reference image is greater than or equal to the maximum value of the reference image, there is no more reference image to allocate, so the reference image index allocation process is terminated, and the process proceeds to step S209 in FIG. Return.
- the reference image index is alternately assigned to the reference image in the time direction and the reference image in the view direction.
- a small reference image index is assigned to a reference image having a high reference ratio, so that the coding efficiency can be improved.
- the types of patterns are not limited to two types.
- four types of allocation patterns (methods) may be prepared, and one of the four types may be designated as a default on the encoding side.
- the B picture shown in the center represents a picture to be encoded.
- Pictures shown in the left-right direction with respect to the picture to be encoded represent reference pictures (reference pictures in the time direction) having different times in the same view.
- POC indicates an index of time (picture output order) and indicates that the time has passed in the order of A to E.
- a thick solid line arrow indicates the index assignment order for List0
- a thick dotted line arrow indicates the index assignment order for List1.
- a small reference image index is assigned to List0 in descending order of POC among the images that can be referred to.
- a small reference image index is assigned to List1 in the ascending order of POC among the images that can be referred to.
- a time first (time direction priority) pattern is shown.
- time ⁇ first pattern as shown in the example of FIG. 12, after assigning an index to a reference picture having a different time in the same view (which can be referred to in the time direction), a reference to a different view in the same time (view direction) This is a pattern in which an index is assigned to a picture that can be referred to.
- the index of refPicList0 [0] is assigned to the B picture with ViewID X and POC B, and the index of refPicList0 [1] is ViewID X Thus, it is assigned to a B picture whose POC is A. Thereafter, the index of refPicList0 [2] is assigned to the B picture whose ViewID is W and POC is C, and the index of refPicList0 [3] is assigned to the B picture whose ViewID is V and POC is C.
- the index of refPicList1 [0] is assigned to the B picture with ViewID X and POC D
- the index of refPicList1 [1] is ViewID X
- it is assigned to a B picture whose POC is E.
- the index of refPicList1 [2] is assigned to the B picture whose ViewID is Y and POC is C
- the index of refPicList1 [3] is assigned to the B picture whose ViewID is Z and POC is C.
- a view fist (view direction priority) pattern is shown.
- the view fist pattern as shown in the example of FIG. 13, after assigning an index to a reference picture of a different view at the same time (which can be referred to in the view direction), a reference having a different time in the same view (time direction) This is a pattern in which an index is assigned to a picture that can be referred to.
- the index of refPicList0 [0] is assigned to the B picture with ViewID W and POC C, and the index of refPicList0 [1] is ViewID V Thus, it is assigned to a B picture whose POC is C. Thereafter, the index of refPicList0 [2] is assigned to the B picture whose ViewID is X and POC is B, and the index of refPicList0 [3] is assigned to the B picture whose ViewID is X and POC is A.
- the index of refPicList1 [0] is assigned to the B picture with ViewID Y and POC C
- the index of refPicList1 [1] is ViewID Z
- it is assigned to a B picture whose POC is C.
- the index of refPicList1 [2] is assigned to the B picture whose ViewID is X and POC is D
- the index of refPicList1 [3] is assigned to the B picture whose ViewID is X and POC is E.
- a time first zig-zag (alternating time direction priority) pattern is shown.
- the time first zig-zag pattern is a reference picture that can be referred to in the same view at different times (can be referred to in the time direction) and a reference of a view that is different in the same time (can be referenced in the view direction ) A pattern in which indexes are alternately assigned to pictures.
- index assignment is started from reference pictures having different times in the same view.
- the index of refPicList0 [0] is assigned to the B picture with ViewID X and POC B, and then the index of refPicList0 [1] is ViewID Is assigned to a B picture with W and POC C.
- the index of refPicList0 [2] is assigned to the B picture whose ViewID is X and POC is A, and then the index of refPicList0 [3] is the B picture whose ViewID is V and POC is C Assigned.
- the index of refPicList1 [0] is assigned to the B picture with ViewID X and POC D, and then the index of refPicList1 [1] is ViewID Is assigned to a B picture whose Y is PO and POC is C.
- the index of refPicList1 [2] is assigned to the B picture whose ViewID is X and POC is E, and then the index of refPicList1 [3] is the B picture whose ViewID is Z and POC is C Assigned.
- a view first zig-zag (alternating view direction priority) pattern is shown.
- the view ⁇ first zig-zag pattern is a reference (referenced in the view direction) of a different view at the same time and a reference (referenceable in the time direction) of the same view. )
- the index of refPicList0 [0] is assigned to the B picture with ViewID W and POC C, and then the index of refPicList0 [1] is ViewID Is assigned to a B picture with X and POC B.
- the index of refPicList0 [2] is assigned to the B picture whose ViewID is V and POC is C, and then the index of refPicList0 [3] is the B picture whose ViewID is X and POC is A Assigned.
- the index of refPicList1 [0] is assigned to the B picture with ViewID Y and POC C, and then the index of refPicList1 [1] is ViewID Is assigned to a B picture whose X is PO and whose POC is D.
- the index of refPicList1 [2] is assigned to the B picture whose ViewID is Z and POC is C, and then the index of refPicList1 [3] is the B picture whose ViewID is X and POC is E Assigned.
- the above four types of allocation patterns are specified in the following syntax in the image encoding device 100. For example, it is specified (added) by the syntax processing unit 121.
- FIG. 16 is a diagram illustrating an example of syntax of a sequence parameter set (SPS). The number at the left end of each line is the line number given for explanation.
- default_reference_picture_index_pattern_idc is set in the 13th line.
- the default_reference_picture_index_pattern_idc is a pattern ID indicating which allocation pattern is designated as a default of the reference picture index in this sequence.
- Default_reference_picture_index_pattern_idc selects the reference picture index initialization process as follows.
- default_reference_picture_index_pattern_idc 0
- default_reference_picture_index_pattern_idc 1
- default_reference_picture_index_pattern_idc 2
- default_reference_picture_index_pattern_idc 3
- the default assignment pattern is specified for each sequence in the sequence parameter set.
- the above-mentioned pattern ID is provided in the picture parameter set (PPS) which is an existing NAL unit, and the default assignment pattern is set at each scene change or at the start and end of a fade as described below. It may be switched according to.
- PPS picture parameter set
- FIG. 17 shows a table showing four types of allocation patterns and what scenes are assumed to be set as defaults when assumed.
- the assumed scene is a scene whose temporal correlation is significantly higher than the inter-view correlation
- 0 is described in the pattern ID of the picture parameter set on the encoding side
- the default allocation pattern is time.
- First pattern is set.
- the assumed scene is a scene in which the correlation between views is considerably higher than the temporal correlation
- 1 is described in the pattern ID of the picture parameter set on the encoding side, and the default allocation pattern is The view first pattern is set.
- the assumed scene is a scene where the temporal correlation is about the same as the inter-view correlation, but the temporal correlation is relatively higher than the inter-view correlation, on the encoding side. It is set like this. That is, 2 is described in the pattern ID of the picture parameter set, and a time first zig-zag pattern is set as a default allocation pattern.
- the assumed scene is a scene where the temporal correlation is about the same as the inter-view correlation, but the temporal correlation is relatively higher than the inter-view correlation, on the encoding side. It is set like this. That is, 2 is described in the pattern ID of the picture parameter set, and a time first zig-zag pattern is set as a default allocation pattern.
- the default allocation pattern is switched.
- the default assignment pattern can be switched at every scene change where the scene changes at once, or at the start and end of a fade where the brightness gradually changes.
- a small reference image index is assigned to a reference image having a high reference ratio, encoding efficiency can be further improved.
- the picture parameter set which is an existing NAL unit, has been described as having the above-described pattern ID.
- the present invention is not limited to PPS, and can be applied to other NAL units as long as it can be applied to multiple pictures. You may make it provide.
- a new NAL unit such as APS (Adaptation Parameter Set) may be provided.
- bit amount is increased by sending the pattern ID for each of a plurality of pictures in the picture parameter set, compared with sending for each of the sequence parameters described above. Therefore, when an increase in the amount of bits does not become a load, it is desirable to send a pattern ID for each of a plurality of pictures using a picture parameter set.
- a default allocation pattern is selected from four types of patterns.
- a default allocation pattern is selected using any one of the four types of patterns described above (two types, etc.) as candidates. You may make it choose.
- the number of candidate patterns is not limited to four, and may be any number.
- the pattern flag and the example of the pattern ID have been described.
- the pattern flag and the pattern ID are both examples of pattern identification information for identifying a pattern.
- a pattern flag and a pattern ID are included.
- FIG. 18 illustrates a configuration of another embodiment of an image encoding device as an image processing device to which the present disclosure is applied.
- the image encoding device 311 shown in FIG. 18 encodes image data using a prediction process by the HEVC method.
- the image encoding device 311 includes an A / D (Analog / Digital) conversion unit 321, a screen rearrangement buffer 322, a calculation unit 323, an orthogonal transformation unit 324, a quantization unit 325, and a lossless encoding unit 326. And a storage buffer 327.
- the image encoding device 311 includes an inverse quantization unit 328, an inverse orthogonal transform unit 329, a calculation unit 330, a deblocking filter 331, a frame memory 332, a selection unit 333, an intra prediction unit 334, a motion prediction / compensation unit 335, A predicted image selection unit 336 and a rate control unit 337 are included.
- the image encoding device 311 includes an adaptive offset filter 341 and an adaptive loop filter 342 between the deblocking filter 331 and the frame memory 332.
- the A / D conversion unit 321 performs A / D conversion on the input image data, outputs it to the screen rearrangement buffer 322, and stores it.
- the screen rearrangement buffer 322 rearranges the stored frame images in the display order in the order of frames for encoding according to the GOP (Group of Picture) structure.
- the screen rearrangement buffer 322 supplies the image with the rearranged frame order to the arithmetic unit 323.
- the screen rearrangement buffer 322 also supplies the image in which the order of the frames is rearranged to the intra prediction unit 334 and the motion prediction / compensation unit 335.
- the calculation unit 323 subtracts the prediction image supplied from the intra prediction unit 334 or the motion prediction / compensation unit 335 via the prediction image selection unit 336 from the image read from the screen rearrangement buffer 322, and the difference information Is output to the orthogonal transform unit 324.
- the calculation unit 323 subtracts the predicted image supplied from the intra prediction unit 334 from the image read from the screen rearrangement buffer 322.
- the arithmetic unit 323 subtracts the predicted image supplied from the motion prediction / compensation unit 335 from the image read from the screen rearrangement buffer 322.
- the orthogonal transform unit 324 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 323, and supplies the transform coefficient to the quantization unit 325.
- the quantization unit 325 quantizes the transform coefficient output from the orthogonal transform unit 324.
- the quantization unit 325 supplies the quantized transform coefficient to the lossless encoding unit 326.
- the lossless encoding unit 326 performs lossless encoding such as variable length encoding and arithmetic encoding on the quantized transform coefficient.
- the lossless encoding unit 326 acquires parameters such as information indicating the intra prediction mode from the intra prediction unit 334, and acquires parameters such as information indicating the inter prediction mode and motion vector information from the motion prediction / compensation unit 335.
- the lossless encoding unit 326 encodes the quantized transform coefficient, encodes each acquired parameter (syntax element), and makes it a part of the header information of the encoded data (multiplexes).
- the lossless encoding unit 326 supplies the encoded data obtained by encoding to the accumulation buffer 327 for accumulation.
- lossless encoding processing such as variable length encoding or arithmetic encoding is performed.
- variable length coding include CAVLC (Context-Adaptive Variable Length Coding).
- arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).
- the accumulation buffer 327 temporarily holds the encoded stream (data) supplied from the lossless encoding unit 326 and records, for example, a recording (not shown) in the subsequent stage as an encoded image encoded at a predetermined timing. Output to devices and transmission lines. That is, the accumulation buffer 327 is also a transmission unit that transmits the encoded stream.
- the transform coefficient quantized by the quantization unit 325 is also supplied to the inverse quantization unit 328.
- the inverse quantization unit 328 inversely quantizes the quantized transform coefficient by a method corresponding to the quantization by the quantization unit 325.
- the inverse quantization unit 328 supplies the obtained transform coefficient to the inverse orthogonal transform unit 329.
- the inverse orthogonal transform unit 329 performs inverse orthogonal transform on the supplied transform coefficient by a method corresponding to the orthogonal transform processing by the orthogonal transform unit 324.
- the inverse orthogonal transformed output (restored difference information) is supplied to the calculation unit 330.
- the calculation unit 330 is supplied from the intra prediction unit 334 or the motion prediction / compensation unit 335 to the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 329, that is, the restored difference information, via the predicted image selection unit 336. Predicted images are added to obtain a locally decoded image (decoded image).
- the calculation unit 330 adds the prediction image supplied from the intra prediction unit 334 to the difference information.
- the calculation unit 330 adds the prediction image supplied from the motion prediction / compensation unit 335 to the difference information.
- the decoded image as the addition result is supplied to the deblocking filter 331 and the frame memory 332.
- the deblocking filter 331 removes block distortion of the decoded image by appropriately performing deblocking filter processing.
- the deblocking filter 331 supplies the filter processing result to the adaptive offset filter 341.
- the adaptive offset filter 341 performs an offset filter (SAO: Sample adaptive offset) process that mainly removes ringing on the image after filtering by the deblocking filter 331.
- SAO Sample adaptive offset
- the adaptive offset filter 341 uses the quad-tree structure in which the type of the offset filter is determined for each divided region and the offset value for each divided region, and performs filtering on the image after filtering by the deblocking filter 331. Apply processing.
- the quad-tree structure and the offset value for each divided region are calculated by the adaptive offset filter 341 and used.
- the calculated quad-tree structure and the offset value for each divided region are encoded by the lossless encoding unit 326 and transmitted to the image decoding device 351 shown in FIG.
- the adaptive offset filter 341 supplies the filtered image to the adaptive loop filter 342.
- the adaptive loop filter 342 performs an adaptive loop filter (ALF) process in units of LCU, which is the maximum encoding unit, as an ALF processing unit.
- ALF adaptive loop filter
- the filtered image is supplied to the frame memory 332.
- a two-dimensional Wiener filter is used as a filter.
- filters other than the Wiener filter may be used.
- the adaptive loop filter 342 performs filter processing on the image after filtering by the adaptive offset filter 341 for each LCU using the filter coefficient, and supplies the filter processing result to the frame memory 332.
- the filter coefficient is calculated and used by the adaptive loop filter 342 so as to minimize the residual from the original image from the screen rearrangement buffer 312 for each LCU.
- the calculated filter coefficient is encoded by the lossless encoding unit 326 and transmitted to the image decoding device 351 in FIG.
- the processing unit of the adaptive loop filter 342 is not limited to this.
- the frame memory 332 outputs the stored reference image to the intra prediction unit 334 or the motion prediction / compensation unit 335 via the selection unit 333 at a predetermined timing.
- the frame memory 332 supplies the reference image to the intra prediction unit 334 via the selection unit 333.
- the frame memory 332 supplies the reference image to the motion prediction / compensation unit 335 via the selection unit 333.
- the selection unit 333 supplies the reference image to the intra prediction unit 334 when the reference image supplied from the frame memory 332 is an image on which intra coding is performed.
- the selection unit 333 supplies the reference image to the motion prediction / compensation unit 335 when the reference image supplied from the frame memory 332 is an image to be inter-encoded.
- the intra prediction unit 334 performs intra prediction (intra-screen prediction) that generates a predicted image using pixel values in the screen.
- the intra prediction unit 334 performs intra prediction in a plurality of modes (intra prediction modes).
- the intra prediction unit 334 generates predicted images in all intra prediction modes, evaluates each predicted image, and selects an optimal mode. When the intra prediction unit 334 selects the optimal intra prediction mode, the intra prediction unit 334 supplies the predicted image generated in the optimal mode to the calculation unit 323 and the calculation unit 330 via the predicted image selection unit 336.
- the intra prediction unit 334 supplies parameters such as intra prediction mode information indicating the adopted intra prediction mode to the lossless encoding unit 326 as appropriate.
- the motion prediction / compensation unit 335 uses the input image supplied from the screen rearrangement buffer 322 and the reference image supplied from the frame memory 332 via the selection unit 333 for the image to be inter-coded. Perform motion prediction. In addition, the motion prediction / compensation unit 335 performs motion compensation processing according to the motion vector detected by motion prediction, and generates a predicted image (inter-predicted image information).
- the motion prediction / compensation unit 335 performs inter prediction processing in all candidate inter prediction modes, and generates a prediction image.
- the motion prediction / compensation unit 335 supplies the generated predicted image to the calculation unit 323 and the calculation unit 330 via the predicted image selection unit 336.
- the motion prediction / compensation unit 335 supplies parameters such as inter prediction mode information indicating the employed inter prediction mode and motion vector information indicating the calculated motion vector to the lossless encoding unit 326.
- the predicted image selection unit 336 supplies the output of the intra prediction unit 334 to the calculation unit 323 and the calculation unit 330 in the case of an image to be subjected to intra coding, and in the case of an image to be subjected to inter coding, the motion prediction / compensation unit 335.
- the output is supplied to the calculation unit 323 and the calculation unit 330.
- the rate control unit 337 controls the rate of the quantization operation of the quantization unit 325 based on the compressed image stored in the storage buffer 327 so that overflow or underflow does not occur.
- FIG. 19 illustrates a configuration of an embodiment of an image decoding device as an image processing device to which the present disclosure is applied.
- An image decoding device 351 shown in FIG. 19 is a decoding device corresponding to the image encoding device 311 in FIG. 18, and decodes encoded data from the image encoding device 311 by the HEVC method.
- the encoded data encoded by the image encoding device 311 is transmitted to the image decoding device 351 corresponding to the image encoding device 311 via a predetermined transmission path and decoded.
- the image decoding device 351 includes a storage buffer 361, a lossless decoding unit 362, an inverse quantization unit 363, an inverse orthogonal transform unit 364, a calculation unit 365, a deblocking filter 366, a screen rearrangement buffer 367, And a D / A converter 368. Further, the image decoding device 351 includes a frame memory 369, a selection unit 370, an intra prediction unit 371, a motion prediction / compensation unit 372, and a selection unit 373.
- the image decoding device 351 includes an adaptive offset filter 381 and an adaptive loop filter 382 between the deblocking filter 366 and the screen rearrangement buffer 367 and the frame memory 369.
- the accumulation buffer 361 is also a receiving unit that receives transmitted encoded data.
- the accumulation buffer 361 receives and accumulates the transmitted encoded data.
- This encoded data is encoded by the image encoding device 311.
- the lossless decoding unit 362 decodes the encoded data read from the accumulation buffer 361 at a predetermined timing by a method corresponding to the encoding method of the lossless encoding unit 326 in FIG.
- the lossless decoding unit 362 supplies parameters such as information indicating the decoded intra prediction mode to the intra prediction unit 371, and supplies parameters such as information indicating the inter prediction mode and motion vector information to the motion prediction / compensation unit 372. . Further, the lossless decoding unit 362 supplies the decoded parameters (such as filter coefficients) of the adaptive loop filter to the adaptive loop filter 382. The lossless decoding unit 362 supplies the decoded adaptive offset parameters (such as the quad-tree structure and the offset value for each divided region) to the adaptive offset filter 381.
- the inverse quantization unit 363 inversely quantizes the coefficient data (quantization coefficient) obtained by decoding by the lossless decoding unit 362 by a method corresponding to the quantization method of the quantization unit 325 in FIG. That is, the inverse quantization unit 363 performs inverse quantization of the quantization coefficient using the quantization parameter supplied from the image encoding device 311 in the same manner as the inverse quantization unit 328 in FIG.
- the inverse quantization unit 363 supplies the inversely quantized coefficient data, that is, the orthogonal transform coefficient, to the inverse orthogonal transform unit 364.
- the inverse orthogonal transform unit 364 is a method corresponding to the orthogonal transform method of the orthogonal transform unit 324 in FIG. 18, performs inverse orthogonal transform on the orthogonal transform coefficient, and converts it into residual data before being orthogonally transformed by the image encoding device 311. Corresponding decoding residual data is obtained.
- the decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 365. Further, a prediction image is supplied to the calculation unit 365 from the intra prediction unit 371 or the motion prediction / compensation unit 372 via the selection unit 373.
- the computing unit 365 adds the decoded residual data and the predicted image, and obtains decoded image data corresponding to the image data before the predicted image is subtracted by the computing unit 323 of the image encoding device 311.
- the computing unit 365 supplies the decoded image data to the deblocking filter 366.
- the deblocking filter 366 removes block distortion of the decoded image by appropriately performing deblocking filter processing.
- the deblocking filter 366 supplies the filter processing result to the adaptive offset filter 381.
- the adaptive offset filter 381 performs an offset filter (SAO) process that mainly removes ringing on the image after filtering by the deblocking filter 366.
- SAO offset filter
- the adaptive offset filter 381 uses a quad-tree structure in which the type of the offset filter is determined for each divided region and the offset value for each divided region, and applies a filter to the image after filtering by the deblocking filter 366. Apply processing.
- the adaptive offset filter 381 supplies the filtered image to the adaptive loop filter 382.
- the quad-tree structure and the offset value for each divided region are calculated by the adaptive offset filter 341 of the image encoding device 311, encoded and sent.
- the quad-tree structure encoded by the image encoding device 311 and the offset value for each divided region are received by the image decoding device 351, decoded by the lossless decoding unit 362, and used by the adaptive offset filter 381. .
- the adaptive loop filter 382 is basically configured in the same manner as the adaptive loop filter 342 in FIG. 18, and performs an adaptive loop filter (ALF: Adaptive Loop Filter) process in units of LCUs, which are the maximum coding units, as an ALF processing unit. .
- the adaptive loop filter 382 performs filter processing on the image after filtering by the adaptive offset filter 381 using the filter coefficient for each LCU, and supplies the filter processing result to the frame memory 369 and the screen rearrangement buffer 367. .
- the filter coefficient is calculated for each LUC by the adaptive loop filter 342 of the image encoding device 311 and is encoded and sent by the lossless decoding unit 62.
- the screen rearrangement buffer 367 rearranges images. That is, the order of frames rearranged for the encoding order by the screen rearrangement buffer 322 in FIG. 18 is rearranged in the original display order.
- the D / A conversion unit 368 performs D / A conversion on the image supplied from the screen rearrangement buffer 367, and outputs and displays the image on a display (not shown).
- the output of the adaptive loop filter 382 is further supplied to the frame memory 369.
- the frame memory 369, the selection unit 370, the intra prediction unit 371, the motion prediction / compensation unit 372, and the selection unit 373 are the frame memory 332, the selection unit 333, the intra prediction unit 334, and the motion prediction / compensation unit of the image encoding device 311. 335 and the predicted image selection unit 336, respectively.
- the selection unit 370 reads the inter-processed image and the referenced image from the frame memory 369 and supplies the image to the motion prediction / compensation unit 372. In addition, the selection unit 370 reads an image used for intra prediction from the frame memory 369 and supplies the image to the intra prediction unit 371.
- the intra prediction unit 371 is appropriately supplied from the lossless decoding unit 362 with information indicating the intra prediction mode obtained by decoding the header information. Based on this information, the intra prediction unit 371 generates a prediction image from the reference image acquired from the frame memory 369 and supplies the generated prediction image to the selection unit 373.
- Information obtained by decoding header information is supplied from the lossless decoding unit 362 to the motion prediction / compensation unit 372.
- the motion prediction / compensation unit 372 generates a prediction image from the reference image acquired from the frame memory 369 based on the information supplied from the lossless decoding unit 362, and supplies the generated prediction image to the selection unit 373.
- the selection unit 373 selects the prediction image generated by the motion prediction / compensation unit 372 or the intra prediction unit 371 and supplies the selected prediction image to the calculation unit 365.
- the encoding method is H.264.
- the H.264 / AVC format and the HEVC format are used as a base, but the present disclosure is not limited to this, and other encoding schemes / decoding schemes that perform motion parallax prediction and compensation can be applied.
- the present disclosure includes, for example, MPEG, H.264, and the like.
- image information such as discrete cosine transformation and motion compensation, such as 26x
- network media such as satellite broadcasting, cable television, the Internet, or mobile phones.
- the present invention can be applied to an image encoding device and an image decoding device used in
- the present disclosure can be applied to an image encoding device and an image decoding device that are used when processing on a storage medium such as an optical disk, a magnetic disk, and a flash memory.
- the present disclosure can also be applied to motion prediction / compensation devices included in those image encoding devices and image decoding devices.
- the series of processes described above can be executed by hardware or can be executed by software.
- a program constituting the software is installed in the computer.
- the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.
- a CPU (Central Processing Unit) 501 of a computer 500 has various programs according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 513 into a RAM (Random Access Memory) 503. Execute the process.
- the RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.
- the CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504.
- An input / output interface 510 is also connected to the bus 504.
- the input / output interface 510 includes an input unit 511 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), an output unit 512 including a speaker, and a hard disk.
- a communication unit 514 including a storage unit 513 and a modem is connected. The communication unit 514 performs communication processing via a network including the Internet.
- a drive 515 is connected to the input / output interface 510 as necessary, and a removable medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 513 as necessary.
- a removable medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 513 as necessary.
- a program constituting the software is installed from a network or a recording medium.
- the recording medium is distributed to distribute the program to the user separately from the apparatus main body, and includes a magnetic disk (including a flexible disk) on which the program is recorded, an optical disk ( It only consists of removable media 521 consisting of CD-ROM (compact disc -read only memory), DVD (including digital Versatile disc), magneto-optical disk (including MD (mini disc)), or semiconductor memory. Rather, it is composed of a ROM 502 on which a program is recorded and a hard disk included in the storage unit 513, which is distributed to the user in a state of being incorporated in the apparatus main body in advance.
- a magnetic disk including a flexible disk
- an optical disk It only consists of removable media 521 consisting of CD-ROM (compact disc -read only memory), DVD (including digital Versatile disc), magneto-optical disk (including MD (mini disc)), or semiconductor memory. Rather, it is composed of a ROM 502 on which a program is recorded and a hard disk included in the storage unit 513, which is distributed
- the program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.
- the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.
- system represents the entire apparatus composed of a plurality of devices (apparatuses).
- the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units).
- the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit).
- a configuration other than that described above may be added to the configuration of each device (or each processing unit).
- a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). . That is, the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present technology.
- An image encoding device and an image decoding device include a transmitter or a receiver in optical broadcasting, satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication
- the present invention can be applied to various electronic devices such as a recording device that records an image on a medium such as a magnetic disk and a flash memory, or a playback device that reproduces an image from these storage media.
- a recording device that records an image on a medium such as a magnetic disk and a flash memory
- a playback device that reproduces an image from these storage media.
- FIG. 21 illustrates an example of a schematic configuration of a television device to which the above-described embodiment is applied.
- the television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, And a bus 912.
- Tuner 902 extracts a signal of a desired channel from a broadcast signal received via antenna 901, and demodulates the extracted signal. Then, the tuner 902 outputs the encoded bit stream obtained by the demodulation to the demultiplexer 903. In other words, the tuner 902 serves as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.
- the demultiplexer 903 separates the video stream and audio stream of the viewing target program from the encoded bit stream, and outputs each separated stream to the decoder 904. Further, the demultiplexer 903 extracts auxiliary data such as EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control unit 910. Note that the demultiplexer 903 may perform descrambling when the encoded bit stream is scrambled.
- EPG Electronic Program Guide
- the decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. Then, the decoder 904 outputs the video data generated by the decoding process to the video signal processing unit 905. In addition, the decoder 904 outputs audio data generated by the decoding process to the audio signal processing unit 907.
- the video signal processing unit 905 reproduces the video data input from the decoder 904 and causes the display unit 906 to display the video.
- the video signal processing unit 905 may cause the display unit 906 to display an application screen supplied via a network.
- the video signal processing unit 905 may perform additional processing such as noise removal on the video data according to the setting.
- the video signal processing unit 905 may generate a GUI (Graphical User Interface) image such as a menu, a button, or a cursor, and superimpose the generated image on the output image.
- GUI Graphic User Interface
- the display unit 906 is driven by a drive signal supplied from the video signal processing unit 905, and displays a video on a video screen of a display device (for example, a liquid crystal display, a plasma display, or an OELD (Organic ElectroLuminescence Display) (organic EL display)). Or an image is displayed.
- a display device for example, a liquid crystal display, a plasma display, or an OELD (Organic ElectroLuminescence Display) (organic EL display)). Or an image is displayed.
- the audio signal processing unit 907 performs reproduction processing such as D / A conversion and amplification on the audio data input from the decoder 904, and outputs audio from the speaker 908.
- the audio signal processing unit 907 may perform additional processing such as noise removal on the audio data.
- the external interface 909 is an interface for connecting the television apparatus 900 to an external device or a network.
- a video stream or an audio stream received via the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also has a role as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.
- the control unit 910 includes a processor such as a CPU and memories such as a RAM and a ROM.
- the memory stores a program executed by the CPU, program data, EPG data, data acquired via a network, and the like.
- the program stored in the memory is read and executed by the CPU when the television device 900 is activated, for example.
- the CPU controls the operation of the television device 900 according to an operation signal input from the user interface 911 by executing the program.
- the user interface 911 is connected to the control unit 910.
- the user interface 911 includes, for example, buttons and switches for the user to operate the television device 900, a remote control signal receiving unit, and the like.
- the user interface 911 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 910.
- the bus 912 connects the tuner 902, the demultiplexer 903, the decoder 904, the video signal processing unit 905, the audio signal processing unit 907, the external interface 909, and the control unit 910 to each other.
- the decoder 904 has the function of the image decoding apparatus according to the above-described embodiment. Thereby, the encoding efficiency in the multi-view encoding can be improved when the image is decoded by the television device 900.
- FIG. 22 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied.
- a mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, a control unit 931, an operation A portion 932 and a bus 933.
- the antenna 921 is connected to the communication unit 922.
- the speaker 924 and the microphone 925 are connected to the audio codec 923.
- the operation unit 932 is connected to the control unit 931.
- the bus 933 connects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the demultiplexing unit 928, the recording / reproducing unit 929, the display unit 930, and the control unit 931 to each other.
- the mobile phone 920 has various operation modes including a voice call mode, a data communication mode, a shooting mode, and a videophone mode, and is used for sending and receiving voice signals, sending and receiving e-mail or image data, taking images, and recording data. Perform the action.
- the analog voice signal generated by the microphone 925 is supplied to the voice codec 923.
- the audio codec 923 converts an analog audio signal into audio data, A / D converts the compressed audio data, and compresses it. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922.
- the communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal.
- the communication unit 922 demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923.
- the audio codec 923 decompresses the audio data and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.
- the control unit 931 generates character data constituting the e-mail in response to an operation by the user via the operation unit 932.
- the control unit 931 causes the display unit 930 to display characters.
- the control unit 931 generates e-mail data in response to a transmission instruction from the user via the operation unit 932, and outputs the generated e-mail data to the communication unit 922.
- the communication unit 922 encodes and modulates email data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921.
- the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal.
- the communication unit 922 demodulates and decodes the received signal to restore the email data, and outputs the restored email data to the control unit 931.
- the control unit 931 displays the content of the electronic mail on the display unit 930 and stores the electronic mail data in the storage medium of the recording / reproducing unit 929.
- the recording / reproducing unit 929 has an arbitrary readable / writable storage medium.
- the storage medium may be a built-in storage medium such as RAM or flash memory, and is externally mounted such as a hard disk, magnetic disk, magneto-optical disk, optical disk, USB (Unallocated Space Space Bitmap) memory, or memory card. It may be a storage medium.
- the camera unit 926 images a subject to generate image data, and outputs the generated image data to the image processing unit 927.
- the image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the storage / playback unit 929.
- the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and the multiplexed stream is the communication unit 922. Output to.
- the communication unit 922 encodes and modulates the stream and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921.
- the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal.
- These transmission signal and reception signal may include an encoded bit stream.
- the communication unit 922 demodulates and decodes the received signal to restore the stream, and outputs the restored stream to the demultiplexing unit 928.
- the demultiplexing unit 928 separates the video stream and the audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923.
- the image processing unit 927 decodes the video stream and generates video data.
- the video data is supplied to the display unit 930, and a series of images is displayed on the display unit 930.
- the audio codec 923 decompresses the audio stream and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.
- the image processing unit 927 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Accordingly, encoding efficiency in multi-viewpoint encoding can be improved when encoding and decoding images with the mobile phone 920.
- FIG. 23 shows an example of a schematic configuration of a recording / reproducing apparatus to which the above-described embodiment is applied.
- the recording / reproducing device 940 encodes audio data and video data of a received broadcast program and records the encoded data on a recording medium.
- the recording / reproducing device 940 may encode audio data and video data acquired from another device and record them on a recording medium, for example.
- the recording / reproducing device 940 reproduces data recorded on the recording medium on a monitor and a speaker, for example, in accordance with a user instruction. At this time, the recording / reproducing device 940 decodes the audio data and the video data.
- the recording / reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface. 950.
- Tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Then, the tuner 941 outputs the encoded bit stream obtained by the demodulation to the selector 946. That is, the tuner 941 has a role as a transmission unit in the recording / reproducing apparatus 940.
- the external interface 942 is an interface for connecting the recording / reproducing apparatus 940 to an external device or a network.
- the external interface 942 may be, for example, an IEEE1394 interface, a network interface, a USB interface, or a flash memory interface.
- video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission unit in the recording / reproducing device 940.
- the encoder 943 encodes video data and audio data when the video data and audio data input from the external interface 942 are not encoded. Then, the encoder 943 outputs the encoded bit stream to the selector 946.
- the HDD 944 records an encoded bit stream in which content data such as video and audio are compressed, various programs, and other data on an internal hard disk. Further, the HDD 944 reads out these data from the hard disk when reproducing video and audio.
- the disk drive 945 performs recording and reading of data to and from the mounted recording medium.
- the recording medium mounted on the disk drive 945 is, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.) or a Blu-ray (registered trademark) disk. It may be.
- the selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video and audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 during video and audio reproduction.
- the decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. The decoder 904 outputs the generated audio data to an external speaker.
- OSD 948 reproduces the video data input from the decoder 947 and displays the video. Further, the OSD 948 may superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.
- the control unit 949 includes a processor such as a CPU and memories such as a RAM and a ROM.
- the memory stores a program executed by the CPU, program data, and the like.
- the program stored in the memory is read and executed by the CPU when the recording / reproducing apparatus 940 is activated, for example.
- the CPU controls the operation of the recording / reproducing apparatus 940 in accordance with an operation signal input from the user interface 950, for example, by executing the program.
- the user interface 950 is connected to the control unit 949.
- the user interface 950 includes, for example, buttons and switches for the user to operate the recording / reproducing device 940, a remote control signal receiving unit, and the like.
- the user interface 950 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.
- the encoder 943 has the function of the image encoding apparatus according to the above-described embodiment.
- the decoder 947 has the function of the image decoding apparatus according to the above-described embodiment.
- FIG. 24 illustrates an example of a schematic configuration of an imaging apparatus to which the above-described embodiment is applied.
- the imaging device 960 images a subject to generate an image, encodes the image data, and records it on a recording medium.
- the imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a bus. 972.
- the optical block 961 is connected to the imaging unit 962.
- the imaging unit 962 is connected to the signal processing unit 963.
- the display unit 965 is connected to the image processing unit 964.
- the user interface 971 is connected to the control unit 970.
- the bus 972 connects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970 to each other.
- the optical block 961 includes a focus lens and a diaphragm mechanism.
- the optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962.
- the imaging unit 962 includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and converts an optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. Then, the imaging unit 962 outputs the image signal to the signal processing unit 963.
- CCD Charge-Coupled Device
- CMOS Complementary Metal-Oxide Semiconductor
- the signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962.
- the signal processing unit 963 outputs the image data after the camera signal processing to the image processing unit 964.
- the image processing unit 964 encodes the image data input from the signal processing unit 963 and generates encoded data. Then, the image processing unit 964 outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes encoded data input from the external interface 966 or the media drive 968 to generate image data. Then, the image processing unit 964 outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may display the image by outputting the image data input from the signal processing unit 963 to the display unit 965. Further, the image processing unit 964 may superimpose display data acquired from the OSD 969 on an image output to the display unit 965.
- the OSD 969 generates a GUI image such as a menu, a button, or a cursor, and outputs the generated image to the image processing unit 964.
- the external interface 966 is configured as a USB input / output terminal, for example.
- the external interface 966 connects the imaging device 960 and a printer, for example, when printing an image.
- a drive is connected to the external interface 966 as necessary.
- a removable medium such as a magnetic disk or an optical disk is attached to the drive, and a program read from the removable medium can be installed in the imaging device 960.
- the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has a role as a transmission unit in the imaging device 960.
- the recording medium mounted on the media drive 968 may be any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory.
- a recording medium may be fixedly mounted on the media drive 968, and a non-portable storage unit such as an internal hard disk drive or an SSD (Solid State Drive) may be configured.
- the control unit 970 includes a processor such as a CPU and memories such as a RAM and a ROM.
- the memory stores a program executed by the CPU, program data, and the like.
- the program stored in the memory is read and executed by the CPU when the imaging device 960 is activated, for example.
- the CPU controls the operation of the imaging device 960 according to an operation signal input from the user interface 971 by executing the program.
- the user interface 971 is connected to the control unit 970.
- the user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960.
- the user interface 971 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 970.
- the image processing unit 964 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Accordingly, encoding efficiency in multi-viewpoint encoding can be improved when encoding and decoding an image in the imaging device 960.
- pattern identification information such as a pattern flag or pattern ID is multiplexed in an encoded stream and transmitted from the encoding side to the decoding side
- the method for transmitting such information is not limited to such an example.
- these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream.
- the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream).
- the information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or the bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.
- this technique can also take the following structures.
- a decoding unit that decodes a bitstream and generates an image;
- An index assigning unit that alternately assigns an index of a reference image referred to when predicting the image generated by the decoding unit to an image that can be referred to in the view direction and an image that can be referred to in the time direction;
- An image processing apparatus comprising: a prediction unit that predicts the image generated by the decoding unit with reference to a reference image of the index allocated by the index allocation unit.
- the index assignment unit repeats a process of assigning an index of the reference image in order of an image that can be referred to in a time direction and an image that can be referred to in a view direction.
- the index assigning unit assigns an index of the reference image to an image that can be referred to in the time direction, and then assigns a time direction priority pattern to an image that can be referred to in the view direction, and the index of the reference image in the view direction.
- the image processing apparatus according to (1), wherein one of the view direction priority patterns to be assigned to an image that can be referred to in the time direction after being assigned to a referenceable image is selected and an index of the reference image is assigned.
- the index allocation unit repeats the process of allocating the index of the reference image in the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction, and the index of the reference image in the view direction.
- the view direction priority pattern that repeats the process of allocating images that can be referred to in the order of images that can be referenced in the time direction is selected, and a pattern to which the reference image is allocated is selected, and an index of the reference image is allocated.
- Image processing apparatus. (8) The image processing device according to (6), wherein the index allocation unit allocates an index of the reference image according to pattern identification information for identifying whether to use the time direction priority pattern or the view direction priority pattern. . (9) The index assignment unit assigns an index of the reference image to an image that can be referred to in the view direction, based on view reference information of a sequence parameter set. Any one of (1) to (8) An image processing apparatus according to 1.
- the index allocating unit determines the reference image based on a decoding order in the case of a P picture and based on a POC (Picture Order Count) in the case of a B picture.
- the image processing apparatus according to any one of (1) to (9).
- the index assigning unit repeats the process of assigning the index of the reference image from an image that can be referred to in the time direction and assigning the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction.
- a view direction priority alternating pattern that repeats the process of assigning an index of the reference image from an image that can be referred to in the view direction and assigning an image that can be referred to in the view direction to an image that can be referred to in the time direction.
- the image processing apparatus according to (1), wherein a pattern to which an index of the reference image is assigned is selected and an index of the reference image is assigned.
- the image processing apparatus Decrypt the image, Decode the bitstream to generate an image, An index of a reference image referred to when predicting the generated image is alternately assigned to an image that can be referred to in the view direction and an image that can be referred to in the time direction.
- an index assigning unit that alternately assigns an index of a reference image referred to when predicting an image to an image that can be referred to in the view direction and an image that can be referred to in the time direction;
- a prediction unit that predicts the image with reference to a reference image of an index allocated by the index allocation unit, and an encoding unit that generates a bitstream by encoding the image predicted by the prediction unit.
- An image processing apparatus (15) The image processing device according to (14), wherein the index assigning unit repeats a process of assigning an index of the reference image in order of an image that can be referred to in a time direction and an image that can be referred to in a view direction.
- the index assigning unit assigns an index of the reference image to an image that can be referred to in the time direction, and then assigns a temporal direction priority pattern to an image that can be referred to in the view direction, and the index of the reference image in the view direction.
- the image processing device according to (14), wherein one of the view direction priority patterns to be assigned to an image that can be referred to in the time direction after being assigned to a referenceable image is selected and an index of the reference image is assigned.
- the index assignment unit repeats the process of assigning the index of the reference image in the order of the image that can be referred to in the time direction to the image that can be referred to in the view direction, and the index of the reference image in the view direction.
- the view direction priority pattern that repeats the process of assigning images that can be referred to in the order of images that can be referred to in the time direction is selected, and a pattern to which the reference image is assigned is selected and an index of the reference image is assigned.
- Image processing apparatus. (21) Pattern identification information for identifying whether the bit stream uses the time direction priority pattern selected by the index allocation unit or the view direction priority pattern, and the bit stream generated by the encoding unit The image processing apparatus according to (19), further including: a transmission unit that transmits.
- the index assignment unit assigns an index of the reference image to an image that can be referred to in the view direction, based on view reference information of a sequence parameter set. Any one of (14) to (21) An image processing apparatus according to 1.
- the index allocation unit determines the reference image based on a decoding order in the case of a P picture and based on a POC (Picture Order Count) in the case of a B picture.
- the image processing apparatus according to any one of (14) to (22).
- the index allocation unit repeats the process of allocating the index of the reference image from an image that can be referred to in the time direction and repeating the process of allocating the image that can be referred to in the time direction to the image that can be referred to in the view direction.
- a view direction priority alternating pattern that repeats the process of assigning an index of the reference image from an image that can be referred to in the view direction and assigning an image that can be referred to in the view direction to an image that can be referred to in the time direction.
- a transmission unit that transmits pattern identification information for identifying whether to use the temporal direction-priority alternating pattern or the view direction-priority alternating pattern and the bitstream generated by the encoding unit is further provided.
- the image processing apparatus An index of a reference image that is referred to when predicting an image is alternately assigned to an image that can be referred to in the view direction and an image that can be referred to in the time direction. Predicting the image with reference to the reference image of the assigned index, An image processing method for generating a bitstream by encoding using the image predicted by the prediction unit.
Abstract
Description
1.第1の実施の形態(画像符号化装置)
2.第2の実施の形態(画像復号装置)
3.第3の実施の形態(4種類のデフォルトの割り当てパターンの例)
4.第4の実施の形態(画像符号化装置)
5.第5の実施の形態(画像復号装置)
6.第6の実施の形態(コンピュータ)
7.応用例
[画像符号化装置の構成例]
図1は、本開示を適用した画像処理装置としての画像符号化装置の一実施の形態の構成を表している。
H.264/AVC方式およびHEVC方式においては、以下のように、同一ビュー内の参照画像インデックスの割り当て方法が規定されている。
図3の例においては、ビュー0、ビュー2、ビュー1の順に復号される場合が示されている。
次に、以上のような画像符号化装置100により実行される各処理の流れについて説明する。最初に、図7のフローチャートを参照して、符号化処理の流れの例を説明する。
次に、図8のフローチャートを参照して、図7のステップS113において実行される参照画像インデックスの割り当て処理の流れの例を説明する。なお、この処理は、復号画像(すなわち、予測対象の画像)がPピクチャまたはBピクチャのみに実行される。
[画像復号装置]
図9は、本開示を適用した画像処理装置としての画像復号装置の一実施の形態の構成を表している。図9に示される画像復号装置200は、図1の画像符号化装置100に対応する復号装置である。
次に、以上のような画像復号装置200により実行される各処理の流れについて説明する。最初に、図10のフローチャートを参照して、復号処理の流れの例を説明する。
次に、図11のフローチャートを参照して、図10のステップS209において実行される参照画像インデックスの割り当て処理の流れの例を説明する。なお、この処理は、復号画像がPピクチャまたはBピクチャのみに実行される。また、図11の参照画像インデックス割り当て処理は、図8の参照画像インデックス割り当て処理と基本的に同様の処理であるので、図8の処理に対する効果やなお書きは、繰り返しになるので記載しないが、図11の処理にも当てはまるものとする。
[4種類の割り当てパターン]
次に、図12乃至図15を参照して、デフォルトとして指定可能な4種類の割り当てパターンについて説明する。図12乃至図15の例において、中央に示されるBピクチャは、符号化対象のピクチャを表している。符号化対象のピクチャに対して左右方向に示されるピクチャは、同一ビューの時間が異なる参照ピクチャ(時間方向の参照画像)を表している。POCは、時刻(ピクチャの出力順序)のインデックスを示し、A乃至Eの順に時刻が経過していることを示している。
図16は、シーケンスパラメータセット(SPS)のシンタックスの例を示す図である。各行の左端の数字は説明のために付した行番号である。
図17は、4種類の割り当てパターンと、どのようなシーンが想定される場合に、それぞれがデフォルトとして設定されるかを示す表を示している。
[画像符号化装置の構成例]
図18は、本開示を適用した画像処理装置としての画像符号化装置の他の実施の形態の構成を表している。
[画像復号装置の構成例]
図19は、本開示を適用した画像処理装置としての画像復号装置の一実施の形態の構成を表している。図19に示される画像復号装置351は、図18の画像符号化装置311に対応する復号装置であり、画像符号化装置311からの符号化データに対して、HEVC方式により復号を行う。
[コンピュータ]
上述した一連の処理は、ハードウエアにより実行することもできるし、ソフトウエアにより実行することもできる。一連の処理をソフトウエアにより実行する場合には、そのソフトウエアを構成するプログラムが、コンピュータにインストールされる。ここで、コンピュータには、専用のハードウエアに組み込まれているコンピュータや、各種のプログラムをインストールすることで、各種の機能を実行することが可能な汎用のパーソナルコンピュータなどが含まれる。
[第1の応用例:テレビジョン受像機]
図21は、上述した実施形態を適用したテレビジョン装置の概略的な構成の一例を示している。テレビジョン装置900は、アンテナ901、チューナ902、デマルチプレクサ903、デコーダ904、映像信号処理部905、表示部906、音声信号処理部907、スピーカ908、外部インタフェース909、制御部910、ユーザインタフェース911、及びバス912を備える。
図22は、上述した実施形態を適用した携帯電話機の概略的な構成の一例を示している。携帯電話機920は、アンテナ921、通信部922、音声コーデック923、スピーカ924、マイクロホン925、カメラ部926、画像処理部927、多重分離部928、記録再生部929、表示部930、制御部931、操作部932、及びバス933を備える。
図23は、上述した実施形態を適用した記録再生装置の概略的な構成の一例を示している。記録再生装置940は、例えば、受信した放送番組の音声データ及び映像データを符号化して記録媒体に記録する。また、記録再生装置940は、例えば、他の装置から取得される音声データ及び映像データを符号化して記録媒体に記録してもよい。また、記録再生装置940は、例えば、ユーザの指示に応じて、記録媒体に記録されているデータをモニタ及びスピーカ上で再生する。このとき、記録再生装置940は、音声データ及び映像データを復号する。
図24は、上述した実施形態を適用した撮像装置の概略的な構成の一例を示している。撮像装置960は、被写体を撮像して画像を生成し、画像データを符号化して記録媒体に記録する。
(1) ビットストリームを復号して、画像を生成する復号部と、
前記復号部により生成された前記画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当てるインデックス割り当て部と、
前記インデックス割り当て部により割り当てられたインデックスの参照画像を参照して、前記復号部により生成された前記画像を予測する予測部と
を備える画像処理装置。
(2) 前記インデックス割り当て部は、前記参照画像のインデックスを、時間方向に参照可能な画像、ビュー方向に参照可能な画像の順に割り当てる処理を繰り返す
前記(1)に記載の画像処理装置。
(3) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
前記(2)に記載の画像処理装置。
(4) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像、時間方向に参照可能な画像の順に割り当てる処理を繰り返す
前記(1)に記載の画像処理装置。
(5) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
前記(4)に記載の画像処理装置。
(6) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像に割り当てた後にビュー方向に参照可能な画像に割り当てる時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像に割り当てた後に時間方向に参照可能な画像に割り当てるビュー方向優先パターンとから、一方選択して、前記参照画像のインデックスを割り当てる
前記(1)に記載の画像処理装置。
(7) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先パターンとから、前記参照画像を割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
前記(6)に記載の画像処理装置。
(8) 前記インデックス割り当て部は、前記時間方向優先パターンを用いるか前記ビュー方向優先パターンを用いるかを識別するパターン識別情報に従って、前記参照画像のインデックスを割り当てる
前記(6)に記載の画像処理装置。
(9) 前記インデックス割り当て部は、ビュー方向に参照可能な画像に対しては、シーケンスパラメータセットのビュー参照情報に基づいて、前記参照画像のインデックスを割り当てる
前記(1)乃至(8)のいずれかに記載の画像処理装置。
(10) 前記インデックス割り当て部は、時間方向に参照可能な画像に対しては、Pピクチャの場合、復号順序に基づいて、Bピクチャの場合、POC(Picture Order Count)に基づいて、前記参照画像のインデックスを割り当てる
前記(1)乃至(9)のいずれかに記載の画像処理装置。
(11) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像から割り当てを始めて時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先の交互パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から割り当てを始めてビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先の交互パターンとから、前記参照画像のインデックスを割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
前記(1)に記載の画像処理装置。
(12)
前記インデックス割り当て部は、前記時間方向優先の交互パターンを用いるか前記ビュー方向優先の交互パターンを用いるかを識別するパターン識別情報に従って、前記参照画像のインデックスを割り当てる
前記(11)に記載の画像処理装置。
(13) 画像処理装置が、
画像を復号し、
ビットストリームを復号して、画像を生成し、
生成された前記画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当て、
割り当てられたインデックスの参照画像を参照して、生成された前記画像を予測する
画像処理方法。
(14) 画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当てるインデックス割り当て部と、
前記インデックス割り当て部により割り当てられたインデックスの参照画像を参照して、前記画像を予測する予測部と
前記予測部により予測された前記画像を用いて符号化してビットストリームを生成する符号化部と
を備える画像処理装置。
(15) 前記インデックス割り当て部は、前記参照画像のインデックスを、時間方向に参照可能な画像、ビュー方向に参照可能な画像の順に割り当てる処理を繰り返す
前記(14)に記載の画像処理装置。
(16) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
前記(15)に記載の画像処理装置。
(17) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像、時間方向に参照可能な画像の順に割り当てる処理を繰り返す
前記(14)に記載の画像処理装置。
(18) 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
前記(17)に記載の画像処理装置。
(19) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像に割り当てた後にビュー方向に参照可能な画像に割り当てる時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像に割り当てた後に時間方向に参照可能な画像に割り当てるビュー方向優先パターンとから、一方を選択して、前記参照画像のインデックスを割り当てる
前記(14)に記載の画像処理装置。
(20) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先パターンとから、前記参照画像を割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
前記(19)に記載の画像処理装置。
(21) 前記ビットストリームは、前記インデックス割り当て部により選択された前記時間方向優先パターンを用いるか前記ビュー方向優先パターンを用いるかを識別するパターン識別情報と、前記符号化部により生成されたビットストリームとを伝送する伝送部を
さらに備える前記(19)に記載の画像処理装置。
(22) 前記インデックス割り当て部は、ビュー方向に参照可能な画像に対しては、シーケンスパラメータセットのビュー参照情報に基づいて、前記参照画像のインデックスを割り当てる
前記(14)乃至(21)のいずれかに記載の画像処理装置。
(23) 前記インデックス割り当て部は、時間方向に参照可能な画像に対しては、Pピクチャの場合、復号順序に基づいて、Bピクチャの場合、POC(Picture Order Count)に基づいて、前記参照画像のインデックスを割り当てる
前記(14)乃至(22)のいずれかに記載の画像処理装置。
(24) 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像から割り当てを始めて時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先の交互パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から割り当てを始めてビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先の交互パターンとから、前記参照画像のインデックスを割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
前記(14)に記載の画像処理装置。
(25) 前記時間方向優先の交互パターンを用いるか前記ビュー方向優先の交互パターンを用いるかを識別するパターン識別情報と、前記符号化部により生成されたビットストリームとを伝送する伝送部を
さらに備える前記(24)に記載の画像処理装置。
(26) 画像処理装置が、
画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当て、
割り当てられたインデックスの参照画像を参照して、前記画像を予測し、
前記予測部により予測された前記画像を用いて符号化してビットストリームを生成する
画像処理方法。
Claims (26)
- ビットストリームを復号して、画像を生成する復号部と、
前記復号部により生成された前記画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当てるインデックス割り当て部と、
前記インデックス割り当て部により割り当てられたインデックスの参照画像を参照して、前記復号部により生成された前記画像を予測する予測部と
を備える画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、時間方向に参照可能な画像、ビュー方向に参照可能な画像の順に割り当てる処理を繰り返す
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
請求項2に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像、時間方向に参照可能な画像の順に割り当てる処理を繰り返す
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
請求項4に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像に割り当てた後にビュー方向に参照可能な画像に割り当てる時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像に割り当てた後に、時間方向に参照可能な画像に割り当てるビュー方向優先パターンとから、一方を選択して、前記参照画像のインデックスを割り当てる
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先パターンとから、前記参照画像を割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
請求項6に記載の画像処理装置。 - 前記インデックス割り当て部は、前記時間方向優先パターンを用いるか前記ビュー方向優先パターンを用いるかを識別するパターン識別情報に従って、前記参照画像のインデックスを割り当てる
請求項6に記載の画像処理装置。 - 前記インデックス割り当て部は、ビュー方向に参照可能な画像に対しては、シーケンスパラメータセットのビュー参照情報に基づいて、前記参照画像のインデックスを割り当てる
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、時間方向に参照可能な画像に対しては、Pピクチャの場合、復号順序に基づいて、Bピクチャの場合、POC(Picture Order Count)に基づいて、前記参照画像のインデックスを割り当てる
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像から割り当てを始めて時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先の交互パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から割り当てを始めてビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先の交互パターンとから、前記参照画像のインデックスを割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
請求項1に記載の画像処理装置。 - 前記インデックス割り当て部は、前記時間方向優先の交互パターンを用いるか前記ビュー方向優先の交互パターンを用いるかを識別するパターン識別情報に従って、前記参照画像のインデックスを割り当てる
請求項11に記載の画像処理装置。 - 画像処理装置が、
ビットストリームを復号して、画像を生成し、
生成された前記画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当て、
割り当てられたインデックスの参照画像を参照して、生成された前記画像を予測する
画像処理方法。 - 画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当てるインデックス割り当て部と、
前記インデックス割り当て部により割り当てられたインデックスの参照画像を参照して、前記画像を予測する予測部と
前記予測部により予測された前記画像を用いて符号化してビットストリームを生成する符号化部と
を備える画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、時間方向に参照可能な画像、ビュー方向に参照可能な画像の順に割り当てる処理を繰り返す
請求項14に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
請求項15に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像、時間方向に参照可能な画像の順に割り当てる処理を繰り返す
請求項14に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、ビュー方向に参照可能な画像すべてに対して割り当てた後に、時間方向に参照可能な画像に対して割り当てる
請求項17に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像に割り当てた後にビュー方向に参照可能な画像に割り当てる時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像に割り当てた後に時間方向に参照可能な画像に割り当てるビュー方向優先パターンとから、一方を選択して、前記参照画像のインデックスを割り当てる
請求項14に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先パターンとから、前記参照画像を割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
請求項19に記載の画像処理装置。 - 前記時間方向優先パターンを用いるか前記ビュー方向優先パターンを用いるかを識別するパターン識別情報と、前記符号化部により生成されたビットストリームとを伝送する伝送部を
さらに備える請求項19に記載の画像処理装置。 - 前記インデックス割り当て部は、ビュー方向に参照可能な画像に対しては、シーケンスパラメータセットのビュー参照情報に基づいて、前記参照画像のインデックスを割り当てる
請求項14に記載の画像処理装置。 - 前記インデックス割り当て部は、時間方向に参照可能な画像に対しては、Pピクチャの場合、復号順序に基づいて、Bピクチャの場合、POC(Picture Order Count)に基づいて、前記参照画像のインデックスを割り当てる
請求項14に記載の画像処理装置。 - 前記インデックス割り当て部は、前記参照画像のインデックスを、時間方向に参照可能な画像から割り当てを始めて時間方向に参照可能な画像からビュー方向に参照可能な画像の順に割り当てる処理を繰り返す時間方向優先の交互パターンと、前記参照画像のインデックスをビュー方向に参照可能な画像から割り当てを始めてビュー方向に参照可能な画像から時間方向に参照可能な画像の順に割り当てる処理を繰り返すビュー方向優先の交互パターンとから、前記参照画像のインデックスを割り当てるパターンを選択して、前記参照画像のインデックスを割り当てる
請求項14に記載の画像処理装置。 - 前記時間方向優先の交互パターンを用いるか前記ビュー方向優先の交互パターンを用いるかを識別するパターン識別情報と、前記符号化部により生成されたビットストリームとを伝送する伝送部を
さらに備える請求項24に記載の画像処理装置。 - 画像処理装置が、
画像を予測する際に参照される参照画像のインデックスを、ビュー方向に参照可能な画像と、時間方向に参照可能な画像とに対して交互に割り当て、
割り当てられたインデックスの参照画像を参照して、前記画像を予測し、
前記予測部により予測された前記画像を用いて符号化してビットストリームを生成する
画像処理方法。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013519453A JP5954587B2 (ja) | 2011-06-07 | 2012-05-30 | 画像処理装置および方法 |
US14/118,408 US10021386B2 (en) | 2011-06-07 | 2012-05-30 | Image processing device which predicts an image by referring a reference image of an allocated index |
CN201280026261.XA CN103563364B (zh) | 2011-06-07 | 2012-05-30 | 图像处理设备及方法 |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011127332 | 2011-06-07 | ||
JP2011-127332 | 2011-06-07 | ||
JP2011-252685 | 2011-11-18 | ||
JP2011252685 | 2011-11-18 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012169403A1 true WO2012169403A1 (ja) | 2012-12-13 |
Family
ID=47295969
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/063911 WO2012169403A1 (ja) | 2011-06-07 | 2012-05-30 | 画像処理装置および方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US10021386B2 (ja) |
JP (1) | JP5954587B2 (ja) |
CN (1) | CN103563364B (ja) |
TW (1) | TWI530161B (ja) |
WO (1) | WO2012169403A1 (ja) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104854868A (zh) * | 2012-12-21 | 2015-08-19 | 索尼公司 | 图像处理设备和方法 |
CN104854869A (zh) * | 2012-12-21 | 2015-08-19 | 索尼公司 | 图像处理装置及方法 |
US9350970B2 (en) | 2012-12-14 | 2016-05-24 | Qualcomm Incorporated | Disparity vector derivation |
CN105814890A (zh) * | 2013-12-10 | 2016-07-27 | 佳能株式会社 | 视频编解码器中的句法元素编码所用的方法和设备 |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7295609B2 (en) | 2001-11-30 | 2007-11-13 | Sony Corporation | Method and apparatus for coding image information, method and apparatus for decoding image information, method and apparatus for coding and decoding image information, and system of coding and transmitting image information |
JP5359657B2 (ja) * | 2009-07-31 | 2013-12-04 | ソニー株式会社 | 画像符号化装置および方法、記録媒体、並びにプログラム |
JP5914962B2 (ja) | 2010-04-09 | 2016-05-11 | ソニー株式会社 | 画像処理装置および方法、プログラム、並びに、記録媒体 |
TWI600318B (zh) | 2010-05-18 | 2017-09-21 | Sony Corp | Image processing apparatus and image processing method |
CN105245895B (zh) | 2010-06-04 | 2019-02-26 | 索尼公司 | 图像处理设备和方法 |
WO2012077607A1 (ja) | 2010-12-07 | 2012-06-14 | ソニー株式会社 | 画像処理装置及び画像処理方法 |
ES2753587T3 (es) | 2010-12-07 | 2020-04-13 | Sony Corp | Dispositivo de procesamiento de imágenes y método de procesamiento de imágenes |
JP5741076B2 (ja) | 2010-12-09 | 2015-07-01 | ソニー株式会社 | 画像処理装置及び画像処理方法 |
RU2609094C2 (ru) | 2011-02-10 | 2017-01-30 | Сони Корпорейшн | Устройство и способ обработки изображений |
BR112013022670B1 (pt) | 2011-03-11 | 2022-05-03 | Sony Corporation | Aparelho e método de processamento de imagem |
EP3313074B1 (en) | 2011-06-28 | 2021-09-22 | Sony Group Corporation | Image processing device, image processing method |
JP5907367B2 (ja) | 2011-06-28 | 2016-04-26 | ソニー株式会社 | 画像処理装置および方法、プログラム、並びに記録媒体 |
CN103765903B (zh) * | 2012-01-18 | 2017-09-01 | 松下知识产权经营株式会社 | 图像解码装置、图像编码装置、图像解码方法及图像编码方法 |
CN103957341B (zh) * | 2014-04-30 | 2016-09-28 | 华为软件技术有限公司 | 图片传输的方法及其相关设备 |
JP6626319B2 (ja) * | 2015-11-18 | 2019-12-25 | キヤノン株式会社 | 符号化装置、撮像装置、符号化方法、及びプログラム |
CN105513101B (zh) * | 2015-12-03 | 2018-08-07 | 小米科技有限责任公司 | 图片处理方法及装置 |
EP3466079B1 (en) * | 2016-05-24 | 2023-07-12 | Nokia Technologies Oy | Method and an apparatus and a computer program for encoding media content |
CN106507106B (zh) * | 2016-11-08 | 2018-03-06 | 中国科学技术大学 | 基于参考片的视频帧间预测编码方法 |
CN108600769A (zh) * | 2018-04-16 | 2018-09-28 | 西安万像电子科技有限公司 | 图像传输方法、装置及系统 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006521071A (ja) * | 2003-03-03 | 2006-09-14 | エルジー エレクトロニクス インコーポレーテッド | 参照ピクチャの選択方法 |
JP2009532934A (ja) * | 2006-03-30 | 2009-09-10 | エルジー エレクトロニクス インコーポレイティド | ビデオ信号のデコーディング/エンコーディング方法及び装置 |
WO2012042895A1 (ja) * | 2010-09-30 | 2012-04-05 | パナソニック株式会社 | 立体映像符号化装置、立体映像撮影装置、および立体映像符号化方法 |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7295609B2 (en) | 2001-11-30 | 2007-11-13 | Sony Corporation | Method and apparatus for coding image information, method and apparatus for decoding image information, method and apparatus for coding and decoding image information, and system of coding and transmitting image information |
KR100506864B1 (ko) * | 2002-10-04 | 2005-08-05 | 엘지전자 주식회사 | 모션벡터 결정방법 |
TWI268715B (en) * | 2004-08-16 | 2006-12-11 | Nippon Telegraph & Telephone | Picture encoding method, picture decoding method, picture encoding apparatus, and picture decoding apparatus |
US7671894B2 (en) * | 2004-12-17 | 2010-03-02 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for processing multiview videos for view synthesis using skip and direct modes |
WO2007148906A1 (en) * | 2006-06-19 | 2007-12-27 | Lg Electronics, Inc. | Method and apparatus for processing a vedeo signal |
KR20120027194A (ko) * | 2009-04-21 | 2012-03-21 | 엘지전자 주식회사 | 다시점 비디오 신호 처리 방법 및 장치 |
KR101620067B1 (ko) * | 2009-05-14 | 2016-05-12 | 삼성전자주식회사 | 다시점 영상 압축 장치 및 방법 |
JP5359657B2 (ja) | 2009-07-31 | 2013-12-04 | ソニー株式会社 | 画像符号化装置および方法、記録媒体、並びにプログラム |
JP5344238B2 (ja) | 2009-07-31 | 2013-11-20 | ソニー株式会社 | 画像符号化装置および方法、記録媒体、並びにプログラム |
JP2011087270A (ja) | 2009-09-17 | 2011-04-28 | Victor Co Of Japan Ltd | 画像符号化装置及び画像符号化方法 |
JP2011082683A (ja) * | 2009-10-05 | 2011-04-21 | Sony Corp | 画像処理装置、画像処理方法、及び、プログラム |
JP5914962B2 (ja) | 2010-04-09 | 2016-05-11 | ソニー株式会社 | 画像処理装置および方法、プログラム、並びに、記録媒体 |
TWI600318B (zh) | 2010-05-18 | 2017-09-21 | Sony Corp | Image processing apparatus and image processing method |
CN105245895B (zh) | 2010-06-04 | 2019-02-26 | 索尼公司 | 图像处理设备和方法 |
US20120075436A1 (en) * | 2010-09-24 | 2012-03-29 | Qualcomm Incorporated | Coding stereo video data |
EP2625853A1 (en) * | 2010-10-05 | 2013-08-14 | Telefonaktiebolaget L M Ericsson (PUBL) | Multi-view encoding and decoding technique based on single-view video codecs |
EP2630799A4 (en) * | 2010-10-20 | 2014-07-02 | Nokia Corp | METHOD AND DEVICE FOR VIDEO CODING AND DECODING |
ES2753587T3 (es) | 2010-12-07 | 2020-04-13 | Sony Corp | Dispositivo de procesamiento de imágenes y método de procesamiento de imágenes |
WO2012077607A1 (ja) | 2010-12-07 | 2012-06-14 | ソニー株式会社 | 画像処理装置及び画像処理方法 |
JP5741076B2 (ja) | 2010-12-09 | 2015-07-01 | ソニー株式会社 | 画像処理装置及び画像処理方法 |
RU2609094C2 (ru) | 2011-02-10 | 2017-01-30 | Сони Корпорейшн | Устройство и способ обработки изображений |
BR112013022670B1 (pt) | 2011-03-11 | 2022-05-03 | Sony Corporation | Aparelho e método de processamento de imagem |
JP5907367B2 (ja) | 2011-06-28 | 2016-04-26 | ソニー株式会社 | 画像処理装置および方法、プログラム、並びに記録媒体 |
SG10201610882XA (en) | 2012-06-29 | 2017-02-27 | Sony Corp | Encoding device and encoding method |
-
2012
- 2012-03-30 TW TW101111523A patent/TWI530161B/zh not_active IP Right Cessation
- 2012-05-30 WO PCT/JP2012/063911 patent/WO2012169403A1/ja active Application Filing
- 2012-05-30 US US14/118,408 patent/US10021386B2/en active Active
- 2012-05-30 JP JP2013519453A patent/JP5954587B2/ja active Active
- 2012-05-30 CN CN201280026261.XA patent/CN103563364B/zh active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006521071A (ja) * | 2003-03-03 | 2006-09-14 | エルジー エレクトロニクス インコーポレーテッド | 参照ピクチャの選択方法 |
JP2009532934A (ja) * | 2006-03-30 | 2009-09-10 | エルジー エレクトロニクス インコーポレイティド | ビデオ信号のデコーディング/エンコーディング方法及び装置 |
WO2012042895A1 (ja) * | 2010-09-30 | 2012-04-05 | パナソニック株式会社 | 立体映像符号化装置、立体映像撮影装置、および立体映像符号化方法 |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9350970B2 (en) | 2012-12-14 | 2016-05-24 | Qualcomm Incorporated | Disparity vector derivation |
US9544566B2 (en) | 2012-12-14 | 2017-01-10 | Qualcomm Incorporated | Disparity vector derivation |
JPWO2014097912A1 (ja) * | 2012-12-21 | 2017-01-12 | ソニー株式会社 | 画像処理装置および方法 |
AU2013365149B2 (en) * | 2012-12-21 | 2017-11-23 | Sony Corporation | Image processing device and method |
EP2938080A4 (en) * | 2012-12-21 | 2016-06-22 | Sony Corp | IMAGE PROCESSING APPARATUS AND METHOD |
KR102197557B1 (ko) * | 2012-12-21 | 2020-12-31 | 소니 주식회사 | 화상 처리 장치 및 방법 |
CN104854869A (zh) * | 2012-12-21 | 2015-08-19 | 索尼公司 | 图像处理装置及方法 |
JPWO2014097913A1 (ja) * | 2012-12-21 | 2017-01-12 | ソニー株式会社 | 画像処理装置および方法 |
CN104854868A (zh) * | 2012-12-21 | 2015-08-19 | 索尼公司 | 图像处理设备和方法 |
KR20150097485A (ko) * | 2012-12-21 | 2015-08-26 | 소니 주식회사 | 화상 처리 장치 및 방법 |
RU2649758C2 (ru) * | 2012-12-21 | 2018-04-04 | Сони Корпорейшн | Устройство и способ обработки изображения |
CN104854868B (zh) * | 2012-12-21 | 2018-05-01 | 索尼公司 | 图像处理设备和方法 |
US9992513B2 (en) | 2012-12-21 | 2018-06-05 | Sony Corporation | Image processing efficient transmission or reception of encoded information |
US10375415B2 (en) | 2012-12-21 | 2019-08-06 | Sony Corporation | Image processing apparatus and method |
US10701395B2 (en) | 2012-12-21 | 2020-06-30 | Sony Corporation | Image processing device and method |
US10674155B2 (en) | 2013-12-10 | 2020-06-02 | Canon Kabushiki Kaisha | Method and apparatus for syntax element encoding in video coding and decoding |
CN105814890A (zh) * | 2013-12-10 | 2016-07-27 | 佳能株式会社 | 视频编解码器中的句法元素编码所用的方法和设备 |
Also Published As
Publication number | Publication date |
---|---|
JPWO2012169403A1 (ja) | 2015-02-23 |
US10021386B2 (en) | 2018-07-10 |
CN103563364A (zh) | 2014-02-05 |
CN103563364B (zh) | 2016-03-30 |
US20140086322A1 (en) | 2014-03-27 |
JP5954587B2 (ja) | 2016-07-20 |
TW201251469A (en) | 2012-12-16 |
TWI530161B (zh) | 2016-04-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5954587B2 (ja) | 画像処理装置および方法 | |
US9894355B2 (en) | Image processing apparatus and method thereof | |
AU2013281945B2 (en) | Image processing device and method | |
WO2013002110A1 (ja) | 画像処理装置および方法 | |
US9591329B2 (en) | Image processing device and method for encoding and decoding image | |
US10547864B2 (en) | Image processing device and method | |
WO2013108690A1 (ja) | 画像処理装置および方法 | |
WO2012157538A1 (ja) | 画像処理装置および方法 | |
WO2013031574A1 (ja) | 画像処理装置および方法 | |
WO2014050731A1 (ja) | 画像処理装置および方法 | |
WO2013108688A1 (ja) | 画像処理装置および方法 | |
EP2763412A1 (en) | Image processing device and method | |
WO2013154026A1 (ja) | 画像処理装置および方法 | |
US9930353B2 (en) | Image decoding device and method | |
WO2013005659A1 (ja) | 画像処理装置および方法 | |
WO2012173022A1 (ja) | 画像処理装置および方法 | |
WO2014050732A1 (ja) | 符号化装置および符号化方法、並びに、復号装置および復号方法 | |
JP6508553B2 (ja) | 画像処理装置および方法 | |
WO2014156705A1 (ja) | 復号装置および復号方法、並びに、符号化装置および符号化方法 | |
WO2014156707A1 (ja) | 画像符号化装置および方法、並びに、画像復号装置および方法 | |
WO2014024723A1 (ja) | 画像処理装置および方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201280026261.X Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12796987 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2013519453 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14118408 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12796987 Country of ref document: EP Kind code of ref document: A1 |