HK1237160B - Image processing apparatus and image processing method - Google Patents

Description

Image processing device and image processing method

Technical Field

The present disclosure relates to an image processing device and an image processing method, and more particularly, to an image processing device and an image processing method capable of improving encoding efficiency when performing prediction using correlation within a picture.

Background Art

In recent years, devices that process digital image information and, for the purpose of efficient information transmission and accumulation, compression-encode the images using a coding system that utilizes the redundancy inherent in image information through orthogonal transform and motion compensation, such as discrete cosine transform, have become widespread. Examples of such coding systems include MPEG (Moving Picture Experts Group), H.264, and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as AVC).

Currently, in order to further improve coding efficiency compared to H.264/AVC, JCTVC (Joint Collaborative Team-Video Coding) is standardizing a coding system called HEVC (High Efficiency Video Coding). JCTVC is a joint standardization organization composed of ITU-T (International Telecommunication Union Telecommunication Standardization Sector) and ISO/IEC (International Organization for Standardization/International Electrotechnical Commission).

Furthermore, in HEVC, range extension (HEVC range extension) has been studied to support higher-order formats, for example, images in color difference signal formats such as 4:2:2 and 4:4:4, profiles for picture contents (see, for example, Non-Patent Document 1).

Intra Block Copy (Intra BC) is a coding tool that uses correlation within a frame and performs motion compensation to perform prediction. Intra BC is considered to be a tool that helps improve the coding efficiency of artificial images such as computer graphics and CG images.

However, the Intra BC technology is not adopted in the HEVC Range Extension described above, but is being studied for standardization of the Picture Content Coding (SCC) extension (see, for example, Non-Patent Document 2).

Since December 2014, the standardization of the HEVC SCC extension has been discussing the commonization of Intra BC and inter coding. Commonalization of Intra BC and inter coding requires commonality in the selection of prediction vectors (MV predictors), the transmission of reference image indices (RefIdx), and the transmission of the difference vector (MVD) between the current motion vector and the prediction vector.

Reference List

Non-patent literature

Non-patent document 1: Jill Boyce, et al. "Draft high efficiency video coding (HEVC) version 2, combined format range extensions (RExt), scalability (SHVC), and multi-view (MV-HEVC) extensions", JCTVC-R1013_v6, 2014.10.1

Non-Patent Document 2: Rajan Joshi, et al. “High Efficiency Video Coding (HEVC) Screen Content Coding: Draft 1”, JCTVC-R1005-v3, September 27, 2014

Summary of the Invention

Technical issues

However, the difference vector becomes large and the encoding efficiency decreases in the following case: when encoding the motion vector used in Intra BC, a prediction vector generated with reference to the motion vector used in inter-frame encoding is used.

The present disclosure has been made in view of the circumstances as described above, and can improve encoding efficiency when prediction is performed using correlation within a screen.

Solution to the problem

The image processing device according to the first aspect of the present disclosure is the following image processing device, which includes: a prediction vector generation unit, which, when encoding a current motion vector of a current block used for prediction using correlation within a picture, sets the candidate block to be unavailable when the type of the reference image of the current block and the type of the reference image of the candidate block corresponding to the reference motion vector are different from each other, and generates a prediction vector of the current motion vector by using the reference motion vector, and the reference motion vector is referenced when generating the prediction vector of the current motion vector; and a differential vector generation unit, which generates a differential vector between the current motion vector and the prediction vector generated by the prediction vector generation unit.

The image processing method according to the first aspect of the present disclosure corresponds to the image processing apparatus according to the first aspect of the present disclosure.

In a first aspect of the present disclosure, when encoding a current motion vector of a current block used for prediction using correlation within a picture, in a case where a type of a reference image of the current block and a type of a reference image of a candidate block corresponding to a candidate of the reference motion vector are different from each other, the candidate block is set to be unavailable, a prediction vector of the current motion vector is generated by using the reference motion vector, and a differential vector between the current motion vector and the prediction vector is generated, wherein the reference motion vector is referenced when generating the prediction vector of the current motion vector.

An image processing device according to a second aspect of the present disclosure is an image processing device including: a prediction vector generating section that, when decoding a current motion vector of a current block used for prediction using correlation within a picture, sets the candidate block to be unavailable when a type of a reference image of the current block and a type of a reference image of a candidate block corresponding to a candidate of the reference motion vector are different from each other, and generates a prediction vector of the current motion vector by using the reference motion vector, the reference motion vector being referenced when generating the prediction vector of the current motion vector; and

A motion vector generation section that adds a differential vector between a current motion vector and a prediction vector and the prediction vector generated by the prediction vector generation section to generate a current motion vector.

The image processing method according to the second aspect of the present disclosure corresponds to the image processing device according to the second aspect of the present disclosure.

In a second aspect of the present disclosure, when decoding a current motion vector of a current block used for prediction using correlation within a picture, in a case where the type of a reference image of the current block and the type of a reference image of a candidate block corresponding to a candidate of the reference motion vector are different from each other, the candidate block is set to be unavailable, a prediction vector of the current motion vector is generated by using the reference motion vector, a differential vector between the current motion vector and the prediction vector and the prediction vector are added, and the current motion vector is generated, wherein the reference motion vector is referenced when generating the prediction vector of the current motion vector.

It should be noted that the image processing apparatuses according to the first aspect and the second aspect can be realized by causing a computer to execute a program.

Furthermore, the program executed by a computer in order to realize the image processing apparatus according to the first aspect and the second aspect may be provided by being transmitted via a transmission medium or recorded on a recording medium.

The image processing apparatus according to the first aspect and the second aspect may be an independent apparatus, or may be an internal block constituting one apparatus.

Beneficial effects of the present invention

According to the first aspect of the present disclosure, it is possible to encode an image. In addition, according to the first aspect of the present disclosure, it is possible to improve encoding efficiency when performing prediction using correlation within a screen.

According to the second aspect of the present disclosure, it is possible to decode an encoded stream. In addition, according to the second aspect of the present disclosure, it is possible to decode an encoded stream with improved encoding efficiency when prediction is performed using correlation within a picture.

It should be noted that the effects described herein are not necessarily limited, and any of the effects described herein may be produced.

BRIEF DESCRIPTION OF THE DRAWINGS

[Figure 1] Figure 1 is a diagram used to describe Intra BC.

[Figure 2] Figure 2 is a diagram used to describe the difference between Intra BC and inter-frame coding.

[Figure 3] Figure 3 is a diagram for describing the types of reference images in HEVC.

[Fig. 4] Fig. 4 is a diagram showing an example of setting of candidate blocks in HEVC.

[Fig. 5] Fig. 5 is a diagram showing an example of setting of candidate blocks in HEVC.

[Figure 6] Figure 6 is a diagram for describing candidates for a reference motion vector.

[ Fig. 7] Fig. 7 is a diagram showing an example of setting of candidate blocks in the first embodiment.

[ Fig. 8] Fig. 8 is a diagram showing an example of setting of candidate blocks in the first embodiment.

[ Fig. 9] Fig. 9 is a diagram showing an example of candidates for reference images in HEVC.

[Figure 10] Figure 10 is a diagram for describing the reference image list creation process in HEVC.

[ Fig. 11] Fig. 11 is a diagram showing an example of candidates for a reference image in the first embodiment.

[Fig. 12] Fig. 12 is a diagram for describing the reference image list creation process in the first embodiment.

[ Fig. 13] Fig. 13 is a block diagram showing a configuration example of a first embodiment of an encoding device to which the present disclosure is applied.

[Figure 14] Figure 14 is a block diagram showing a configuration example of the encoding unit of Figure 13.

[Figure 15] Figure 15 is a flowchart for describing the stream generation processing of the encoding device of Figure 13.

[Figure 16] Figure 16 is a flowchart used to describe the details of the encoding process of Figure 15.

[Figure 17] Figure 17 is a flowchart used to describe the details of the encoding process of Figure 15.

[Figure 18] Figure 18 is a flowchart for describing the details of the reference image creation processing of Figure 16.

[Figure 19] Figure 19 is a flowchart used to describe the details of the prediction vector list generation processing of Figure 16.

[Figure 20] Figure 20 is a block diagram showing a configuration example of a first embodiment of a decoding device to which the present disclosure is applied.

[Figure 21] Figure 21 is a block diagram showing a configuration example of the decoding unit of Figure 20.

[Figure 22] Figure 22 is a flowchart used to describe the image generation processing of the decoding device of Figure 20.

[Figure 23] Figure 23 is a flowchart used to describe the details of the decoding process of Figure 22.

[Figure 24] Figure 24 is a diagram used to describe the prediction vector list generation method in the second embodiment.

[Figure 25] Figure 25 is a diagram used to describe the prediction vector list generation method in the second embodiment.

[Figure 26] Figure 26 is a diagram used to describe the prediction vector list generation method in the second embodiment.

[ Fig. 27] Fig. 27 is a diagram showing an example of setting of candidate blocks in the second embodiment.

[Figure 28] Figure 28 is a flowchart for describing the details of the reference image creation process in the second embodiment.

[Figure 29] Figure 29 is a flowchart for describing the details of the prediction vector list generation processing in the second embodiment.

[Figure 30] Figure 30 is a diagram used to describe the reference image list creation processing in the third embodiment.

[Figure 31] Figure 31 is a flowchart for describing the details of the reference image list creation processing in the third embodiment.

[Figure 32] Figure 32 is a block diagram showing a configuration example of computer hardware.

[Figure 33] Figure 33 is a diagram showing a schematic configuration example of a television device to which the present disclosure is applied.

[ Fig. 34] Fig. 34 is a diagram showing a schematic configuration example of a mobile phone to which the present disclosure is applied.

[ Fig. 35] Fig. 35 is a diagram illustrating a schematic configuration example of a recording/reproducing device to which the present disclosure is applied.

[ Fig. 36] Fig. 36 is a diagram illustrating a schematic configuration example of an imaging apparatus to which the present disclosure is applied.

[Figure 37] Figure 37 shows a schematic configuration example of a video set to which the present disclosure is applied.

[Figure 38] Figure 38 shows a schematic configuration example of a video processor to which the present disclosure is applied.

[Figure 39] Figure 39 shows another schematic configuration example of a video processor to which the present disclosure is applied.

DETAILED DESCRIPTION

Hereinafter, the premise of the present disclosure and a mode for carrying out the present disclosure (hereinafter, referred to as an embodiment) will be described. It should be noted that the description will be given in the following order.

0. Premise of the present disclosure (Figures 1 and 2)

1. First Embodiment: Encoding Device and Decoding Device (FIGS. 3 to 23)

2. Second Embodiment: Encoding Device and Decoding Device (FIGS. 24 to 29)

3. Third Embodiment: Encoding Device and Decoding Device (FIGS. 30 and 31)

4. Fourth embodiment: Computer (Figure 32)

5. Fifth embodiment: Television device (FIG. 33)

6. Sixth embodiment: mobile phone (Figure 34)

7. Seventh Embodiment: Recording/Reproducing Device (FIG. 35)

8. Eighth Embodiment: Imaging Device (FIG. 36)

9. Ninth embodiment: Video suite (Figures 37 to 39)

<Prerequisites of this Disclosure>

Description of Intra BC

FIG. 1 is a diagram for describing an intra block copy (Intra BC).

As shown in FIG1 , in Intra BC, the reference image for predicting a PU (Prediction Unit) 11 within an image 10 is the image 10. Therefore, a block 12 that is included in the same image 10 as the PU 11 and has a high correlation with the PU 11 is set as a reference block for the PU 11, and a vector 13 between the PUs 11 and 12 is detected as a motion vector.

(Description of the difference between Intra BC and Inter-frame coding)

FIG. 2 is a diagram for describing the difference between Intra BC and inter-frame coding.

As shown in Figure 2, the precision of motion vectors in intra coding is integer pixel precision, while the precision of motion vectors in inter coding is 1/4 pixel precision. Furthermore, in intra coding, since the coding blocks within the same picture are set as reference blocks, at least one of the lengths in the longitudinal direction and the length in the transverse direction of the motion vector is longer. However, in inter coding, since coding blocks in different pictures are set as reference blocks, the length of the motion vector depends on the motion between pictures.

Intra BC uses the image before filtering as the reference image, but in inter-frame coding, the image after filtering is used as the reference image. Typically, the image before filtering is stored in a cache memory, while the image after filtering is stored in a DRAM (Dynamic Random Access Memory).

As described above, the accuracy and length of motion vectors differ between Intra BC and Inter coding. Therefore, if Intra BC and Inter coding are standardized and a prediction vector generated by referencing an Inter coding motion vector is used when encoding an Intra BC motion vector, the difference vector becomes larger, and coding efficiency decreases. It should be noted that the motion vector referenced when generating the prediction vector is hereinafter referred to as a reference motion vector.

Furthermore, the storage destination for reference images differs between Intra BC and Inter coding. Therefore, when Intra BC and Inter coding are standardized, cache memory and DRAM are used interchangeably as the storage destination for reference images. However, since the decoding device does not know the storage destination for its reference images, it accesses the DRAM in vain. This increases the access bandwidth of the DRAM.

<First embodiment>

(Overview of the First Embodiment)

First, an outline of the first embodiment will be described with reference to FIG. 3 to FIG. 12 .

FIG. 3 is a diagram for describing types of reference pictures in HEVC (High Efficiency Video Coding).

As shown in FIG3, in HEVC, the type of a reference picture with a small difference from the POC (Picture Sequence Number) of the image to be processed currently (hereinafter referred to as the current picture) is set to STRP (Short-Term Reference Picture). In addition, the type of a reference picture with a large difference from the POC of the current picture (Current Pic) is set to LTRP (Long-Term Reference Picture).

As shown in Figures 4 and 5, the candidate block is set to unavailable in the following case: the reference image of the current PU to be processed within the current image (hereinafter, referred to as the current block) and the reference image of the candidate PU corresponding to the reference motion vector (hereinafter, the candidate block) are different in type from each other.

For example, as shown in FIG3 , the candidate block 22 is set to be unavailable in the following case: the type of the reference image of the current block 21 is STRP, and the type of the reference image of the candidate block 22 corresponding to the candidate (PMV) of the reference motion vector of the current motion vector (target MV) is LTRP. Therefore, the motion vector of the candidate block 22 is excluded from the candidate of the reference motion vector and is not used to generate the prediction vector.

It should be noted that the candidate blocks are PUs adjacent to the left, upper left, lower left, upper, and upper right of the current block at the same time of the day, as well as PUs located at the same position as the current block at different times of the day and PUs adjacent to the current block at the lower right of the current block. Hereinafter, in the case of specifically distinguishing between candidate blocks at the same time of the day as the current block and other candidate blocks, candidate blocks at the same time of the day as the current block are referred to as spatial-direction candidate blocks, and in the case of specifically distinguishing between candidate blocks at different times of the day than the current block and other candidate blocks, candidate blocks at different times of the day than the current block are referred to as temporal-direction candidate blocks.

4 and 5, when the type of the reference image of the current block is the same as the type of the reference image of the candidate block, the candidate block is set to be available. Therefore, the motion vector of the candidate block is set as a candidate for the reference motion vector.

In the following case, a prediction vector is generated by scaling the reference motion vector based on the difference in POC between the reference image of the current block and the reference image of the candidate block: when the type of the reference image of the current block and the type of the reference image of the candidate block are both STRP, the motion vector of the candidate block is selected as the reference motion vector.

At the same time, in the following cases, the reference motion vector is set as the prediction vector without change (not scaled): when the type of the reference image of the current block and the type of the reference image of the candidate block are both LTRP, the motion vector of the candidate block is selected as the reference motion vector.

Note that in current HEVC, Intra BC and inter-frame coding are not standardized, so as shown in Figure 5, reference picture types other than LTRP and STRP are not assumed. Therefore, in the following case, the setting of candidate blocks is undefined: the reference picture type is "Intra BC", and "Intra BC" is the type of reference picture used in Intra BC.

When Intra BC and Inter coding are simply commonized, the type of the reference picture used in Intra BC differs from the actual type and is set to STRP. Therefore, as shown in FIG6 , the candidates for the reference motion vector of the current motion vector (target MV) used in Intra BC of the current block 31 include the motion vector (PMV) of the candidate block 32, whose actual reference picture type is STRP.

However, as shown in Figure 2, the precision and length of motion vectors differ between Intra BC and Inter coding. Therefore, if the motion vector of candidate block 32 is selected as the reference motion vector for the current motion vector of current block 31, the difference vector between the current motion vector and the predicted vector becomes larger. Consequently, coding efficiency decreases.

In this regard, in the first embodiment, the type of reference picture used in Intra BC (i.e., the type of reference picture whose actual type is "Intra BC") is set to LTRP. Specifically, the current picture is newly defined as a candidate for a reference picture (if intra_block_copy_enabled_flag is equal to 1, the following applies: RefPicCurr = CurrPic). Then, the newly defined current picture is added as a candidate for a reference picture, and the type of the reference picture is set to LTRP (all reference pictures included in RefPicCurr, RefPicSetLtCurr, and RefPicSetLtFoll are marked as "used for long-term reference.").

Therefore, the current image is added to the reference image list, which is a list of candidates for reference images, and marked as "used for long-term reference." Therefore, the setting of candidate blocks for the current block whose actual reference image type is "Intra BC" is performed similarly to the setting of candidate blocks for the current block whose actual reference image type is LTRP.

Specifically, as shown in Figures 7 and 8, if either the actual type of the reference image of the current block or the actual type of the reference image of the candidate block is "Intra BC" and the other actual type is STRP, the candidate block is set to be unavailable. Therefore, the motion vector of the candidate block is excluded from the candidates for the reference motion vector and is not used to generate the prediction vector.

In addition, as shown in Figures 7 and 8, when both the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are "Intra BC" or LTRP, the candidate block is set to be available. Therefore, the motion vector of the candidate block is set as a candidate for the reference motion vector.

As described above, in a case where either the actual type of the reference image of the current block and the actual type of the reference image of the candidate block is "Intra BC" and the other actual type is STRP, the motion vector of the candidate block is not used to generate a prediction vector, and encoding efficiency is improved.

FIG. 9 is a diagram illustrating an example of candidates for a reference image in HEVC.

As shown in Figure 9, in HEVC, an image that precedes the current image (C) in display order and has a small difference with the POC of the current image (Short-term before Curr) is a candidate for a reference image. In addition, an image that follows the current image (C) in display order and has a small difference with the POC of the current image (Short-term after Curr) is a candidate for a reference image.

In addition, images that precede the current image (C) in display order and have a large difference in POC with the current image (Long-term) are candidates for reference images. In addition, images that are the same as the current image (C) in display order but have different viewpoints, bits, spatial resolutions, S/Ns, frame rates, color difference signal formats, etc. (Inter-layer) are candidates for reference images.

FIG. 10 is a diagram for describing a reference image list creation process for registering candidates for the reference image of FIG. 9 .

As shown in Figure 10, in the reference image list creation process, a separate list is first created for each category of reference image candidates. Information used to identify the reference image candidates (hereinafter referred to as image identification information) is registered in the separate list. In other words, a RefPicSetStCurrBefore list is created, in which the image identification information of each image (short-term before Curr) is registered in ascending order of difference from the POC of the current image.

Similarly, a RefPicSetStCurrAfter list of the image (Short-term after Curr) and a RefPicSetLtCurr list of the image (Long-term) are created. In addition, a RefPicSetIvCurr list is created in which the image identification information of each image in the image (Inter-layer) is registered in a predetermined order.

For each piece of image identification information registered in the RefPicSetStCurrBefore list, the RefPicSetStCurrAfter list, and the RefPicSetLtCurr list, used_by_curr can be set. used_by_curr indicates whether the image identified by the image identification information is used as a candidate for the reference image of the current image. When used_by_curr indicates that the image is used as a candidate for the reference image of the current image, used_by_curr is 1, and when used_by_curr indicates that the image is not used, used_by_curr is 0.

In the example of FIG. 10 , used_by_curr is set to 1 for the first and second pieces of image identification information in the RefPicSetStCurrBefore and RefPicSetStCurrAfter lists and the first piece of image identification information in the RefPicSetLtCurr list.

Next, two temporary lists, namely, RefPicListTemp0 list and RefPicListTemp1 list, are created based on separate lists. RefPicListTemp0 list prioritizes registering candidates for reference images that precede the current image in display order, and RefPicListTemp1 list prioritizes registering candidates for reference images that follow the current image in display order.

Specifically, the RefPicListTemp0 list is created, which sequentially registers image identification information of the RefPicSetStCurrBefore list with used_by_curr set to 1, image identification information registered in the RefPicSetIvCurr list, image identification information of the RefPicSetStCurrAfter list with used_by_curr set to 1, and image identification information of the RefPicSetLtCurr list with used_by_curr set to 1.

Furthermore, a RefPicListTemp1 list is created which sequentially registers image identification information of the RefPicSetStCurrAfter list with used_by_curr set to 1, image identification information of the RefPicSetStCurrBefore list with used_by_curr set to 1, image identification information of the RefPicSetLtCurr list with used_by_curr set to 1, and image identification information registered in the RefPicSetIvCurr list.

Next, the order of the image identification information registered in the RefPicListTemp0 list and the RefPicListTemp1 list is changed as needed (refer to reordering).

Finally, a reference picture list RefPicList0 is created in which a predetermined number (five in the example of FIG. 10 ) of image identification information from the head registered in the list RefPicListTemp0 are registered in order from the head.

Furthermore, a reference image list RefPicList1 is created in which a predetermined number (four in the example of FIG. 10 ) of image identification information from the head registered in the list RefPicListTemp1 are registered in order from the head.

It should be noted that a value obtained by subtracting 1 from the number of image identification information registered in the reference picture list RefPicList0 (RefPicList1) is arranged in the slice header as num_ref_idx_l0_active_minus1 (num_ref_idx_l1_active_minus1).

As shown in FIG. 11 , when Intra BC and inter-frame coding are standardized, the current picture (CurrPic), which is a reference picture in IntraBC, must be added as a candidate for the reference picture.

In the first embodiment, as described above, since the current image is defined as a candidate for a reference image, a current image list (RefPicCurr) in which the image identification information of the current image is registered can be created in the reference image list creation process. Therefore, the image identification information of the current image is registered in the reference image list.

However, when Intra BC and inter-frame coding are simply unified, it is unknown at which position in the reference picture list the picture identification information of the current picture is registered.

Furthermore, as shown in Figure 2, the storage locations for reference images differ between Intra BC and Inter coding. Therefore, to avoid wasted DRAM access, when the decoding device obtains the index of image identification information for a reference image within the reference image list from the encoding device, it must identify whether the reference image is for Intra BC or Inter coding. In other words, the decoding device must identify whether the reference image is for the current image or a different image.

In this regard, in the first embodiment, as shown in FIG. 12 , the image identification information of the current image registered in the current image list is registered in the beginnings of two reference image lists (RefPicListTemp0 list and RefPicListTemp1 list).

Therefore, when the decoding device obtains the index at the beginning of the reference picture list from the encoding device, the decoding device can recognize that the reference picture is the current picture, that is, the reference picture in the intra BC. Therefore, the decoding device can read the current picture as the reference picture from the cache memory without accessing the DRAM.

It should be noted that only the image identification information of the current image may be registered in the list RefPicList0. In addition, the position where the image identification information of the current image is registered may be set to any predetermined position determined in advance, such as the end, other than the beginning.

(Configuration Example of First Embodiment of Encoding Device)

FIG. 13 is a block diagram showing a configuration example of a first embodiment of an encoding device as an image processing device to which the present disclosure is applied.

An encoding device 50 of FIG. 13 includes a setting section 51 , an encoding section 52 , and a transmission section 53 , and encodes an image by a system according to HEVC.

The unit of coding is a coding unit (CU) having a recursive hierarchical structure. Specifically, a CU is set by dividing an image into CTUs (Coding Tree Units) of a fixed size and dividing the CTU into two any number of times in the horizontal and vertical directions. The maximum size of a CU is the LCU (Largest Coding Unit), and the minimum size of a CU is the SCU (Smallest Coding Unit). In addition, a CU is divided into PUs or TUs (Transform Units).

The setting unit 51 of the encoding device 50 sets parameter sets such as SPS (Sequence Parameter Set), PPS (Picture Parameter Set), VUI (Video Usable Information), and SEI (Supplemental Enhancement Information). The SPS includes information specifying the sizes of LCU and SCU, reference picture list information for creating a reference picture list, and the like.

The reference picture list information is made up of, for example, information specifying picture identification information registered in the separate list and the current picture list, used_by_curr, etc. The setting section 51 supplies the set parameter set to the encoding section 52 .

The encoding unit 52 receives an image in frames. It encodes the input image using the HEVC system. The parameter set supplied from the setup unit 51 is used as needed. The encoding unit 52 generates an encoded stream based on the encoded data obtained from the encoding and the parameter set supplied from the setup unit 51, and supplies the encoded stream to the transmission unit 53.

The transmission section 53 transmits the encoded stream supplied from the encoding section 52 to a decoding device to be described later.

(Configuration Example of Encoding Unit)

FIG14 is a block diagram showing a configuration example of the encoding section 52 of FIG13 .

The encoding section 52 of FIG14 includes an A/D conversion section 71, a screen sorting buffer 72, a calculation section 73, an orthogonal transform section 74, a quantization section 75, a lossless encoding section 76, an accumulation buffer 77, a generation section 78, an inverse quantization section 79, an inverse orthogonal transform section 80, and an addition section 81. Furthermore, the encoding section 52 includes a filter 82, a frame memory 85, a switch 86, an intra-frame prediction section 87, a list creation section 88, a motion prediction/compensation section 89, a prediction vector generation section 90, a difference vector generation section 91, a prediction picture selection section 92, and a rate control section 93.

The A/D conversion unit 71 of the encoding unit 52 performs A/D conversion on the frame-by-frame image input as the encoding target, and outputs the converted image as a digital signal to the screen sorting buffer 72 to store it in the screen sorting buffer 72.

The picture sorting buffer 72 sorts the pictures stored in the display order in units of frames into the encoding order according to the GOP (Group of Picture) structure. The picture sorting buffer 72 outputs each of the sorted pictures as the current picture to the calculation section 73, the intra prediction section 87, and the motion prediction/compensation section 89.

The calculation unit 73 performs encoding by subtracting the predicted image supplied by the predicted image selection unit 92 from the current image supplied by the screen sorting buffer 72. The calculation unit 73 outputs the resulting image as residual information to the orthogonal transformation unit 74. It should be noted that when the predicted image is not supplied from the predicted image selection unit 92, the calculation unit 73 outputs the current image read from the screen sorting buffer 72 as residual information to the orthogonal transformation unit 74 without changing it.

The orthogonal transform section 74 performs an orthogonal transform on the residual information from the calculation section 73 in units of TUs. The orthogonal transform section 74 supplies an orthogonal transform coefficient obtained as a result of the orthogonal transform to the quantization section 75.

The quantization section 75 quantizes the orthogonal transform coefficient supplied from the orthogonal transform section 74. The quantization section 75 supplies the quantized orthogonal transform coefficient to the lossless encoding section 76.

The lossless encoding section 76 acquires intra-prediction mode information indicating the optimal intra-prediction mode from the intra-prediction section 87. Furthermore, the lossless encoding section 76 acquires inter-prediction mode information indicating the optimal inter-prediction mode, an index of a reference image, and the like from the motion prediction/compensation section 89, and acquires motion vector information from the difference vector generation section 91. Furthermore, the lossless encoding section 76 acquires offset filter information regarding the adaptive offset filter process from the filter 82.

The lossless encoding unit 76 performs lossless encoding such as variable length coding (e.g., CAVLC (Context Adaptive Variable Length Coding), etc.) and arithmetic coding (e.g., CABAC (Context Adaptive Binary Arithmetic Coding), etc.) on the quantized orthogonal transform coefficients supplied from the quantization unit 75.

Furthermore, the lossless encoding section 76 performs lossless encoding on intra prediction mode information or inter prediction mode information, motion vector information, reference image index, offset filter information, and the like as encoding information related to encoding. The lossless encoding section 76 places the lossless encoded encoding information, a predetermined number (num_ref_idx_lo_minus1, num_ref_idx_lo_minus1) obtained by subtracting 1 from the number of registrations in the reference image list, and the like in a slice header or the like.

The lossless encoding section 76 adds a slice header to the orthogonal transform coefficient supplied from the quantization section 75 , and then supplies the resultant data to the accumulation buffer 77 as encoded data.

The accumulation buffer 77 temporarily stores the encoded data supplied from the lossless encoding section 76. Furthermore, the accumulation buffer 77 supplies the stored encoded data to the generation section 78.

The generation section 78 generates an encoded stream from the parameter set supplied from the setting section 51 of FIG. 13 and the encoded data supplied from the accumulation buffer 77 , and supplies the encoded stream to the transmission section 53 of FIG. 13 .

Furthermore, the quantized orthogonal transform coefficients output from the quantization section 75 are also input to the inverse quantization section 79. The inverse quantization section 79 inversely quantizes the orthogonal transform coefficients quantized by the quantization section 75 using a method corresponding to the quantization method of the quantization section 75. The inverse quantization section 79 supplies the orthogonal transform coefficients obtained as a result of the inverse quantization to the inverse orthogonal transform section 80.

The inverse orthogonal transform section 80 performs inverse orthogonal transform on the orthogonal transform coefficients supplied from the inverse quantization section 79 in units of TUs by a method corresponding to the orthogonal transform method of the orthogonal transform section 74 . The inverse orthogonal transform section 80 supplies the resulting residual information to the addition section 81 .

The addition unit 81 adds the residual information supplied from the inverse orthogonal transform unit 80 and the predicted image supplied from the predicted image selection unit 92 to locally decode the current image. It should be noted that, if no predicted image is supplied from the predicted image selection unit 92, the addition unit 81 uses the residual information supplied from the inverse orthogonal transform unit 80 as the decoding result. The addition unit 81 supplies the locally decoded current image to the frame memory 85. Furthermore, the addition unit 81 supplies the current image, with all regions decoded, to the filter 82 as an encoded image.

The filter 82 performs filtering processing on the encoded image supplied from the addition section 81. Specifically, the filter 82 sequentially performs deblocking filtering processing and adaptive offset filtering (SAO (Sample Adaptive Offset)) processing. The filter 82 supplies the encoded image after the filtering processing to the frame memory 85. In addition, the filter 82 supplies information indicating the offset and type of the adaptive offset filtering processing performed to the lossless encoding section 76 as offset filter information.

The frame memory 85 is composed of, for example, a cache memory and a DRAM. The frame memory 85 stores the current image supplied from the adding section 81 in the cache memory, and stores the encoded image supplied from the filter 82 in the DRAM. Pixels adjacent to the current block in the current image stored in the cache memory are set as peripheral pixels and supplied to the intra prediction section 87 via the switch 86.

Furthermore, the current image stored in the cache memory and the encoded image stored in the DRAM are set as reference images and output to the motion prediction/compensation section 89 via the switch 86 .

The intra prediction section 87 performs intra prediction processing on the current block in all candidate intra prediction modes by using peripheral pixels read from the frame memory 85 via the switch 86 .

Furthermore, the intra prediction unit 87 calculates cost function values (details of which will be described later) for all candidate intra prediction modes based on the current image read from the screen sorting buffer 72 and the predicted image generated as a result of the intra prediction process. The intra prediction unit 87 then determines the intra prediction mode with the smallest cost function value as the optimal intra prediction mode.

The intra prediction unit 87 supplies the predicted image generated in the optimal intra prediction mode and the corresponding cost function value to the predicted image selection unit 92. When the predicted image selection unit 92 notifies the intra prediction unit 87 of the selection of the predicted image generated in the optimal intra prediction mode, the intra prediction unit 87 supplies the intra prediction mode information to the lossless encoding unit 76.

It should be noted that the cost function value is also called RD (Rate Distortion) cost and is calculated based on the technique of High Complexity Mode or Low Complexity Mode as determined in JM (Joint Model), which is reference software in the H.264/AVC system. It should be noted that the reference software in the H.264/AVC system is disclosed at http://iphome.hhi.de/suehring/tml/index.htm.

Specifically, in the case of adopting the High Complexity Mode as a technique for calculating a cost function value, all candidate prediction modes are temporarily subjected to processing up to decoding, and a cost function value expressed by the following equation (1) is calculated for each prediction mode.

[Mathematical formula 1]

Cost(Mode)＝D+λ·R…(1)

D represents the difference (distortion) between the original image and the decoded image, R represents the amount of generated code including the orthogonal transform coefficient, and λ represents the Lagrange multiplier given as a function of the quantization parameter QP.

At the same time, when the low complexity mode is adopted as the technology for calculating the cost function value, the generation of the predicted image and the calculation of the code amount of the encoding information are performed for all candidate prediction modes, and the cost function Cost(Mode) represented by the following formula (2) is calculated for each prediction mode.

[Mathematical formula 2]

Cost(Mode)＝D+QPtoQuant(QP)·Header_Bit…(2)

D represents the difference (distortion) between the original image and the predicted image, Header_Bit represents the code amount of the encoding information, and QPtoQuant represents a function given as a function of the quantization parameter QP.

In low complexity mode, only the predicted images for all prediction modes need to be generated, without generating decoded images, which results in a smaller amount of computation.

The list creation section 88 performs the above-described reference image list creation processing, for example, in units of slices, based on the reference image list information contained in the SPS set by the setting section 51 of FIG. 13 , information arranged in the slice header, and the like.

In other words, the list creation unit 88 creates a reference picture list by sequentially creating a separate list, creating a temporary list, and rearranging the order of the image identification information registered in the temporary list based on the reference picture list information, the information arranged in the slice header, etc. The list creation unit 88 then registers the image identification information of the current picture at the beginning of the reference picture list.

Furthermore, the list creation section 88 sets SRTP or LRTP as the reference image type for each piece of image identification information registered in the reference image list. For example, the list creation section 88 (setting section) sets LTRP (for long-term reference) as the reference image type for the image identified by that image identification information. The list creation section 88 stores the reference image list and also supplies the reference image list to the motion prediction/compensation section 89.

The motion prediction/compensation unit 89 performs motion prediction/compensation processing on the current block in all candidate inter-frame prediction modes. Specifically, based on the reference image list supplied from the list creation unit 88, the motion prediction/compensation unit 89 reads an image identified by the image identification information registered in the reference image list from the frame memory 85 via the switch 86 as a candidate for the reference image.

The motion prediction/compensation section 89 detects the motion vector of IntraBC of the current block in all candidate inter prediction modes with integer pixel accuracy based on the current image from the picture sorting buffer 72 and the current image read as a candidate for the reference image. Furthermore, the motion prediction/compensation section 89 detects the motion vector of inter coding of the current block in all candidate inter prediction modes with fractional pixel accuracy based on the current image from the picture sorting buffer 72 and images other than the current image read as a candidate for the reference image.

Then, the motion prediction/compensation section 89 performs compensation processing on the candidate for the reference image based on the detected motion vector, and generates a predicted image. It should be noted that the inter prediction mode is a mode that indicates the size of the current block and the like.

Furthermore, the motion prediction/compensation unit 89 calculates the cost function values of all candidate inter-frame prediction modes and the candidate reference images based on the current image and the predicted image supplied from the screen sorting buffer 72. The motion prediction/compensation unit 89 determines the inter-frame prediction mode with the smallest cost function value as the optimal inter-frame prediction mode. The motion prediction/compensation unit 89 then supplies the minimum cost function value and the corresponding predicted image to the predicted image selection unit 92.

When the predicted image selection unit 92 notifies the motion prediction/compensation unit 89 of the selection of the predicted image generated in the optimal inter-frame prediction mode, the motion prediction/compensation unit 89 determines the motion vector of the current block corresponding to the predicted image as the current motion vector and supplies the motion vector to the difference vector generation unit 91. Furthermore, the motion prediction/compensation unit 89 determines a candidate reference image corresponding to the predicted image as the reference image and supplies an index corresponding to the position of the image identification information of the reference image in the reference image list to the prediction vector generation unit 90 and the lossless encoding unit 76. Furthermore, the motion prediction/compensation unit 89 supplies the inter-frame prediction mode information to the lossless encoding unit 76.

The prediction vector generation unit 90 generates a prediction vector list of the current motion vector for each reference image list in which a reference image is registered, based on the index supplied from the motion prediction/compensation unit 89 .

Specifically, the prediction vector generation unit 90 reads the type of the reference image corresponding to the index from the reference image list stored in the list creation unit 88 based on the index and stores the type. The prediction vector generation unit 90 sets the spatial direction candidate blocks as candidate blocks to be processed in a predetermined order, and sets the temporal direction candidate blocks as candidate blocks to be processed in a predetermined order.

The prediction vector generation section 90 compares the type of the reference image of the candidate block to be processed stored therein with the type of the reference image of the current block read from the list creation section 88. If the types match each other, the prediction vector generation section 90 sets the candidate block to be processed as available. If the types do not match each other, the prediction vector generation section 90 sets the candidate block to be processed as unavailable.

When the candidate block to be processed is set to be available, the prediction vector generation unit 90 generates a prediction vector for the current motion vector by using a candidate reference motion vector for the motion vector of the candidate block to be processed. The prediction vector generation unit 90 then generates a prediction vector list that registers the generated prediction vectors for a predetermined number of spatial candidate blocks and temporal candidate blocks. It should be noted that if the number of generated prediction vectors does not reach the predetermined number, a zero vector or the like is registered in the prediction vector list. The prediction vector generation unit 90 supplies the prediction vector list to the difference vector generation unit 91.

The difference vector generation section 91 generates a difference vector between each of a predetermined number of prediction vectors registered in the prediction vector list supplied from the prediction vector generation section 90 and the current motion vector to perform prediction encoding on the current motion vector. The difference vector generation section 91 supplies the minimum value of the difference vectors and an index corresponding to the position of the corresponding prediction vector in the prediction vector list to the lossless encoding section 76.

Based on the cost function values supplied from the intra prediction unit 87 and the motion prediction/compensation unit 89, the predicted image selection unit 92 determines the mode with the smallest corresponding cost function value among the optimal intra prediction mode and the optimal inter prediction mode as the optimal prediction mode. The predicted image selection unit 92 then supplies the predicted image of the optimal prediction mode to the calculation unit 73 and the addition unit 81. The predicted image selection unit 92 also notifies the intra prediction unit 87 or the motion prediction/compensation unit 89 of the selection of the predicted image of the optimal prediction mode.

The rate control section 93 controls the rate of the quantization operation of the quantization section 75 based on the encoded data accumulated in the accumulation buffer 77 so that overflow or underflow does not occur.

(Description of Processing of Encoding Device)

FIG. 15 is a flowchart for describing a stream generation process of the encoding device 50 of FIG. 13 .

15 , the setting section 51 of the encoding device 50 sets a parameter set. The setting section 51 supplies the set parameter set to the encoding section 52.

In step S12, the encoding section 52 performs encoding processing on the encoding image in units of frames input from the outside by the system according to HEVC. The details of the encoding processing will be described with reference to FIG16 and FIG17 which will be described later.

In step S13 , the generation section 78 ( FIG. 14 ) of the encoding section 52 generates an encoded stream from the parameter set supplied from the setting section 51 and the accumulated encoded data, and supplies the encoded stream to the transmission section 53 .

In step S14 , the transmission section 53 transmits the encoded stream supplied from the setting section 51 to a decoding device to be described later, and then terminates the processing.

16 and 17 are flowcharts for describing details of the encoding process in step S12 of FIG. 15 .

In step S31 of FIG. 16 , the A/D conversion section 71 of the encoding section 52 performs A/D conversion on the image in frame units input as the encoding object, and outputs the image as the converted digital signal to the screen sorting buffer 72 to store it in the screen sorting buffer 72 .

In step S32, the picture sorting buffer 72 sorts the images stored in the display order into the encoding order according to the GOP structure. The picture sorting buffer 72 supplies each of the sorted frame-based images as the current image to the calculation unit 73, the intra-frame prediction unit 87, and the motion prediction/compensation unit 89.

In step S33 , the list creation section 88 performs reference picture list creation processing, for example, in units of slices, based on the reference picture list information contained in the SPS, information arranged in the slice header, etc. Details of the reference picture list creation processing will be described with reference to FIG.

In step S34, the intra prediction unit 87 performs intra prediction processing on the current block in all candidate intra prediction modes using the surrounding pixels read from the frame memory 85 via the switch 86. Furthermore, the intra prediction unit 87 calculates cost function values for all candidate intra prediction modes based on the current image from the screen sorting buffer 72 and the predicted image generated as a result of the intra prediction processing. The intra prediction unit 87 then determines the intra prediction mode with the smallest cost function value as the optimal intra prediction mode. The intra prediction unit 87 supplies the predicted image generated in the optimal intra prediction mode and the corresponding cost function value to the predicted image selection unit 92.

Furthermore, the motion prediction/compensation unit 89 performs motion prediction/compensation processing on the current block in all candidate inter-frame prediction modes based on the reference image list supplied from the list creation unit 88. Furthermore, the motion prediction/compensation unit 89 calculates cost function values and reference images for all candidate inter-frame prediction modes based on the current image supplied from the screen sorting buffer 72 and the predicted image generated as a result of the motion prediction/compensation processing. The motion prediction/compensation unit 89 determines the inter-frame prediction mode with the smallest cost function value as the optimal inter-frame prediction mode. The motion prediction/compensation unit 89 then supplies the smallest cost function value and the corresponding predicted image to the predicted image selection unit 92.

In step S35, the predicted image selection unit 92 determines the mode having the smallest cost function value between the optimal intra prediction mode and the optimal inter prediction mode as the optimal prediction mode based on the cost function values supplied from the intra prediction unit 87 and the motion prediction/compensation unit 89. The predicted image selection unit 92 then supplies the predicted image of the optimal prediction mode to the calculation unit 73 and the addition unit 81.

In step S36, the predicted image selection unit 92 determines whether the optimal prediction mode is the optimal inter prediction mode. If the optimal prediction mode is determined to be the optimal inter prediction mode in step S36, the predicted image selection unit 92 notifies the motion prediction/compensation unit 89 of the selection of the predicted image generated in the optimal inter prediction mode.

In response to the notification, the motion prediction/compensation unit 89 determines the motion vector of the current block corresponding to the predicted image as the current motion vector, and supplies the current motion vector to the difference vector generation unit 91. Furthermore, the motion prediction/compensation unit 89 determines a candidate reference image corresponding to the predicted image as the reference image, and supplies an index corresponding to the position of the image identification information of the reference image in the reference image list to the prediction vector generation unit 90.

In step S37, the prediction vector generation section 90 performs a prediction vector list generation process of generating a prediction vector list of the current motion vector based on the index supplied from the motion prediction/compensation section 89. Details of the prediction vector list generation process will be described later with reference to FIG.

In step S38, the differential vector generation section 91 generates a differential vector between each of a predetermined number of prediction vectors registered in the prediction vector list supplied from the prediction vector generation section 90 and the current motion vector to perform predictive encoding on the current motion vector. Then, the differential vector generation section 91 generates the minimum value of the differential vectors and an index corresponding to the position of the corresponding prediction vector in the prediction vector list as motion vector information.

In step S39, the motion prediction/compensation section 89 supplies the inter prediction mode information and the index of the reference image to the lossless encoding section 76, and the difference vector generation section 91 supplies the motion vector information to the lossless encoding section 76. Then, the process proceeds to step S41.

Meanwhile, if it is determined in step S36 that the optimal prediction mode is not the optimal inter prediction mode, that is, if the optimal prediction mode is the optimal intra prediction mode, the predicted image selection unit 92 notifies the intra prediction unit 87 of the selection of the predicted image generated in the optimal intra prediction mode. In step S40, the intra prediction unit 87 supplies the intra prediction mode information to the lossless encoding unit 76, and the process proceeds to step S41.

In step S41, the calculation unit 73 performs encoding by subtracting the predicted image supplied from the predicted image selection unit 92 from the current image supplied from the screen sorting buffer 72. The calculation unit 73 outputs the obtained image to the orthogonal transformation unit 74 as residual information.

In step S42 , the orthogonal transform section 74 performs orthogonal transform on the residual information from the calculation section 73 in units of TUs, and supplies the resultant orthogonal transform coefficients to the quantization section 75 .

In step S43 , the quantization section 75 quantizes the orthogonal transform coefficient supplied from the orthogonal transform section 74 , and supplies the quantized orthogonal transform coefficient to the lossless encoding section 76 and the inverse quantization section 79 .

In step S44 of FIG. 17 , the inverse quantization section 79 inversely quantizes the quantized coefficients supplied from the quantization section 75 , and supplies the resultant orthogonal transform coefficients to the inverse orthogonal transform section 80 .

In step S45 , the inverse orthogonal transform section 80 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization section 79 in units of TUs, and supplies the resultant residual information to the addition section 81 .

In step S46, the addition unit 81 locally decodes the current image by adding the residual information supplied from the inverse orthogonal transform unit 80 and the predicted image supplied from the predicted image selection unit 92. The addition unit 81 supplies the locally decoded current image to the frame memory 85. Furthermore, the addition unit 81 supplies the current image, in which all regions have been decoded, to the filter 82 as a coded image.

In step S47 , the filter 82 performs a deblocking filtering process on the encoded image supplied from the adding section 81 .

In step S48, the filter 82 performs adaptive offset filtering on the encoded image after deblocking filtering for each LCU. The filter 82 supplies the resulting encoded image to the frame memory 85. In addition, the filter 82 supplies offset filter information to the lossless encoding unit 76 for each LCU.

In step S49, the frame memory 85 stores the current image supplied from the addition section 81 in the cache memory, and stores the encoded image supplied from the filter 82 in the DRAM. Pixels adjacent to the current block in the current image stored in the cache memory are set as peripheral pixels, and are supplied to the intra prediction section 87 via the switch 86. Furthermore, the current image stored in the cache memory and the encoded image stored in the DRAM are set as reference images and are output to the motion prediction/compensation section 89 via the switch 86.

In step S50 , the lossless encoding section 76 performs lossless encoding on the intra prediction mode information or inter prediction mode information, motion vector information, an index of a reference image, offset filter information, and the like as encoding information.

In step S51, the lossless encoding section 76 performs lossless encoding on the quantized orthogonal transform coefficient supplied from the quantization section 75. Then, the lossless encoding section 76 generates encoded data from the encoding information losslessly encoded by the process of step S50 and the losslessly encoded orthogonal transform coefficient, and supplies the encoded data to the accumulation buffer 77.

In step S52 , the accumulation buffer 77 temporarily accumulates the encoded data supplied from the lossless encoding section 76 .

In step S53 , the rate control section 93 controls the rate of the quantization operation of the quantization section 75 based on the encoded data accumulated in the accumulation buffer 77 so that overflow or underflow does not occur.

In step S54, the accumulation buffer 77 outputs the stored encoded data to the generation section 78. Then, the process returns to step S12 of Fig. 15 and proceeds to step S13.

It should be noted that for the purpose of simplifying the description, intra-frame prediction processing and motion prediction/compensation processing are constantly performed in the encoding processing of Figures 16 and 17, but in fact, in some cases, any of the processing is performed depending on the image type, etc.

FIG18 is a flowchart for describing the details of the reference image list creation processing in step S33 of FIG16 .

In step S71 of FIG18 , list creation unit 88 creates four separate lists for each type of reference image candidate: the RefPicSetStCurrBefore list, the RefPicSetStCurrAfter list, the RefPicSetLtCurr list, and the RefPicSetIvCurr list. The image identification information in each separate list specifies the type of reference image identified by that image identification information. Furthermore, used_by_curr is set for each piece of image identification information in the RefPicSetStCurrBefore list, the RefPicSetStCurrAfter list, and the RefPicSetLtCurr list.

In step S72 , the list creation section 88 creates a current image list in which the image identification information of the current image whose type of reference image is set to LTRP is registered.

In step S73 , the list creation section 88 creates two temporary lists, ie, the RefpicListTemp0 list and the RefpicListTemp1 list, based on the image identification information and used_by_curr of the individual lists.

In step S74 , the list creation section 88 rearranges the order of the image identification information in the temporary list based on ref_pic_list_modification arranged in the slice header, and specifies the order of the image identification information in the temporary list.

In step S75 , the list creating section 88 creates, based on each temporary list, a reference image list in which a predetermined number of pieces of image identification information registered in the temporary list are registered in order from the head.

In step S76, the list creation section 88 registers the image identification information of the current image registered in the current image list at the beginning of the reference image list. The process then returns to step S33 of FIG. 16 and proceeds to step S34.

Fig. 19 is a flowchart for describing the details of the prediction vector list generation process in step S37 of Fig. 16. The prediction vector list generation process is performed for each reference image list in which a reference image is registered.

In step S91 of Fig. 19 , the prediction vector generation unit 90 determines a predetermined candidate block as a candidate block to be processed. In the first process of step S91, a predetermined spatial direction candidate block is determined as a candidate block to be processed.

In step S92, the prediction vector generation unit 90 reads the type of the reference image of the current block corresponding to the index from the reference image list based on the index supplied from the motion prediction/compensation unit 89, and stores the type. Then, the prediction vector generation unit 90 determines whether the read type of the reference image of the current block is STRP.

In the event that the type of the reference image of the current block is determined to be STRP in step S92, the process proceeds to step S93. In step S93, the prediction vector generation section 90 determines whether the type of the reference image of the candidate block to be processed stored therein is STRP.

When it is determined in step S93 that the type of the reference image of the candidate block to be processed is STRP, that is, when the type of the reference image of the current block and the type of the reference image of the candidate block are both STRP, the processing proceeds to step S94.

In step S94, the prediction vector generation unit 90 sets the candidate block to be processed as available. In step S95, the prediction vector generation unit 90 performs scaling on the candidate reference motion vector for the motion vector of the candidate block to be processed based on the difference in the POC between the reference image of the current block and the reference image of the candidate block, and generates a prediction vector. The prediction vector generation unit 90 then registers the prediction vector in the prediction vector list, and the process proceeds to step S100.

Meanwhile, in a case where it is determined in step S92 that the type of the reference image of the current block is not STRP, that is, in a case where the type of the reference image of the current block is LTRP, the process proceeds to step S96.

In step S96, similar to the process in step S93, the prediction vector generation section 90 determines whether the type of the reference image of the candidate block to be processed is STRP. In the case where it is determined in step S96 that the type of the reference image of the candidate block to be processed is STRP, that is, in the case where the type of the reference image of the current block is LTRP and the type of the reference image of the candidate block is STRP, the process proceeds to step S97.

In step S97 , the prediction vector generation section 90 sets the candidate block to be processed to be unavailable, and the process proceeds to step S100 .

Meanwhile, if it is determined in step S93 that the type of the reference image of the candidate block to be processed is not STRP, that is, if the type of the reference image of the current block is STRP and the type of the reference image of the candidate block is LTRP, the process proceeds to step S97. Therefore, the candidate block to be processed is set to unavailable.

In addition, when it is determined in step S96 that the type of the reference image of the candidate block to be processed is not STRP, that is, when the type of the reference image of the current block and the type of the reference image of the candidate block are both LTRP, the processing proceeds to step S98.

In step S98, the prediction vector generation unit 90 sets the candidate block to be processed as available. In step S99, the prediction vector generation unit 90 generates a prediction vector without changing the candidate reference motion vector for the motion vector of the candidate block to be processed. The prediction vector generation unit 90 then registers the prediction vector in the prediction vector list, and the process proceeds to step S100.

In step S100, the prediction vector generation unit 90 determines whether a predetermined number of prediction vectors are registered in the prediction vector list. If it is determined in step S100 that the predetermined number of prediction vectors have not been registered, the process returns to step S91. A predetermined spatial candidate block or temporal candidate block that has not yet been determined as a candidate block to be processed is determined as a candidate block to be processed, and subsequent processing is performed.

Meanwhile, in the event that determination is made in step S100 that the predetermined number of prediction vectors are registered, the prediction vector generation section 90 supplies the prediction vector list to the difference vector generation section 91. Then, the process returns to step S37 of FIG16 and proceeds to step S38.

As described above, when encoding the current motion vector used in Intra BC, if the actual type of the reference image of the current block and the actual type of the reference image of the candidate block differ, the encoding device 50 sets the candidate block to be unavailable. Therefore, the motion vector of the candidate block whose actual type of reference image differs from the actual type of the reference image of the current block is not used to generate the prediction vector. As a result, the difference vector becomes smaller, and encoding efficiency is improved.

Furthermore, the encoding device 50 sets the reference image type of the current block, whose actual type is "Intra BC," to LTRP. Therefore, when the actual type of the reference image of the candidate block is STRP, the encoding device 50 sets the candidate block to be unavailable. This improves encoding efficiency when Intra BC and inter-frame coding are standardized, without significantly changing HEVC when Intra BC and inter-frame coding are not standardized.

Furthermore, the encoding device 50 registers the image identification information of the current image in a predetermined position in the reference image list of the current block. Therefore, the decoding device, which will be described later, can determine whether the reference image is the current image based on the index corresponding to the position of the reference image's image identification information in the reference image list. Therefore, when the reference image is the current image, it is possible to prevent unnecessary access to the DRAM.

(Configuration Example of First Embodiment of Decoding Device)

FIG. 20 is a block diagram showing a configuration example of a first embodiment of a decoding device as an image processing device to which the present disclosure is applied, which decodes an encoded stream transmitted from the encoding device 50 of FIG. 13 .

The decoding device 110 of FIG. 20 is composed of a receiving section 111 , an extracting section 112 , and a decoding section 113 .

The receiving section 111 of the decoding device 110 receives the encoded stream transmitted from the encoding device 50 of FIG. 13 , and supplies the encoded stream to the extracting section 112 .

The extraction section 112 extracts a parameter set and encoded data from the encoded stream supplied from the reception section 111 , and supplies the parameter set and encoded data to the decoding section 113 .

The decoding unit 113 decodes the encoded data supplied from the extraction unit 112 using the HEVC system. At this time, the decoding unit 113 also refers to the parameter set supplied from the extraction unit 112 as needed. The decoding unit 113 outputs an image obtained as a result of the decoding.

(Configuration Example of Decoding Unit)

FIG. 21 is a block diagram showing a configuration example of the decoding section 113 of FIG. 20 .

The decoding section 113 of FIG21 includes an accumulation buffer 131, a lossless decoding section 132, an inverse quantization section 133, an inverse orthogonal transform section 134, an addition section 135, a filter 136, and a screen sorting buffer 139. Furthermore, the decoding section 113 includes a D/A conversion section 140, a frame memory 141, a switch 142, an intra-frame prediction section 143, a prediction vector generation section 144, a motion vector generation section 145, a list creation section 146, a motion compensation section 147, and a switch 148.

The accumulation buffer 131 of the decoding section 113 receives the encoded data from the extraction section 112 of Fig. 20 and accumulates the encoded data. The accumulation buffer 131 supplies the accumulated encoded data to the lossless decoding section 132 as encoded data of the current image.

The lossless decoding section 132 performs lossless decoding corresponding to the lossless encoding by the lossless encoding section 76 of FIG. 14 on the encoded data from the accumulation buffer 131, such as variable-length decoding and arithmetic decoding, to obtain quantized orthogonal transform coefficients and encoding information. The lossless decoding section 132 supplies the quantized orthogonal transform coefficients to the inverse quantization section 133. Furthermore, the lossless decoding section 132 supplies intra-frame prediction mode information and the like as encoding information to the intra-frame prediction section 143. The lossless decoding section 132 supplies the reference image index to the prediction vector generation section 144, supplies the motion vector information to the motion vector generation section 145, and supplies the inter-frame prediction mode information and the reference image index to the motion compensation section 147.

Furthermore, the lossless decoding section 132 supplies the intra prediction mode information or the inter prediction mode information as encoding information to the switch 148. The lossless decoding section 132 supplies the offset filter information to the filter 136 as encoding information.

The inverse quantization section 133, the inverse orthogonal transform section 134, the addition section 135, the filter 136, the frame memory 141, the switch 142, the intra prediction section 143, the prediction vector generation section 144, the list creation section 146, and the motion compensation section 147 respectively perform processes similar to those of the inverse quantization section 79, the inverse orthogonal transform section 80, the addition section 81, the filter 82, the frame memory 85, the switch 86, the intra prediction section 87, the prediction vector generation section 90, the list creation section 88, and the motion prediction/compensation section 89. Thus, the image is decoded.

Specifically, the inverse quantization section 133 inversely quantizes the quantized orthogonal transform coefficient supplied from the lossless decoding section 132 , and supplies the resultant orthogonal transform coefficient to the inverse orthogonal transform section 134 .

The inverse orthogonal transform section 134 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization section 133 in units of TUs. The inverse orthogonal transform section 134 supplies residual information obtained as a result of the inverse orthogonal transform to the addition section 135.

The addition unit 135 adds the residual information supplied from the inverse orthogonal transform unit 134 and the predicted image supplied from the switch 148 to locally decode the current image. It should be noted that, if the predicted image is not supplied from the switch 148, the addition unit 135 sets the residual information supplied from the inverse orthogonal transform unit 134 as the decoding result. The addition unit 135 supplies the locally decoded current image obtained as the decoding result to the frame memory 141. Furthermore, the addition unit 135 supplies the current image, in which all regions have been decoded, as a decoded image to the filter 136.

The filter 136 performs filtering on the decoded image supplied from the addition unit 135. Specifically, the filter 136 first performs deblocking filtering on the decoded image. Next, the filter 136 performs adaptive offset filtering on the decoded image after deblocking filtering for each LCU, using the offset indicated by the offset filtering information from the lossless decoding unit 132. The type of adaptive offset filtering is indicated by the offset filtering information. The filter 136 supplies the decoded image after undergoing adaptive offset filtering to the frame memory 141 and the screen sorting buffer 139.

The screen sorting buffer 139 stores the decoded images in units of frames supplied from the filter 136 . The screen sorting buffer 139 sorts the decoded images in units of frames stored in the encoding order into the original display order, and supplies those images to the D/A conversion section 140 .

The D/A conversion section 140 performs D/A conversion on the decoded images in units of frames supplied from the screen sorting buffer 139 , and outputs those images.

The frame memory 141 is composed of, for example, a cache memory and a DRAM. The frame memory 141 stores the current image supplied from the adder 135 in the cache memory, and stores the decoded image supplied from the filter 136 in the DRAM. Pixels adjacent to the current block in the current image stored in the cache memory are set as peripheral pixels and are supplied to the intra prediction unit 143 via the switch 142.

Furthermore, the current image stored in the cache memory or the decoded image stored in the DRAM is set as a reference image and output to the motion compensation section 147 via the switch 142 .

The intra prediction unit 143 performs intra prediction processing on the current block using the surrounding pixels read from the frame memory 141 via the switch 142 in the optimal intra prediction mode indicated by the intra prediction mode information supplied from the lossless decoding unit 132. The intra prediction unit 143 supplies the resulting predicted image to the switch 148.

14 , the prediction vector generation unit 144 generates a prediction vector list for the current motion vector based on the index supplied from the lossless decoding unit 132 and the reference image list stored in the list creation unit 146. The prediction vector generation unit 144 supplies the prediction vector list to the motion vector generation unit 145.

The motion vector generation unit 145 reads a prediction vector corresponding to an index from the prediction vector list supplied by the prediction vector generation unit 144 based on the index in the motion vector information supplied from the lossless decoding unit 132. The motion vector generation unit 145 adds the prediction vector to the difference vector in the motion vector information to perform predictive decoding on the current motion vector. The motion vector generation unit 145 supplies the resulting current motion vector to the motion compensation unit 147.

Similar to the list creation unit 88 of FIG14 , the list creation unit 146 performs reference picture list creation processing based on the reference picture list information contained in the SPS extracted by the extraction unit 112 of FIG20 , the information arranged in the slice header, etc. The list creation unit 146 stores the obtained reference picture list and supplies it to the motion compensation unit 147.

The motion compensation section 147 performs a motion compensation process on the current block based on the inter prediction mode information from the lossless decoding section 132 , the index of the reference image, and the current motion vector from the motion vector generation section 145 .

Specifically, the motion compensation unit 147 reads the image identification information of the reference image index from the image identification information registered in the image list stored in the list creation unit 146. The motion compensation unit 147 reads the reference image identified by the read image identification information from the frame memory 141 via the switch 142. The motion compensation unit 147 performs motion compensation processing on the current block using the reference image and the current motion vector in the optimal inter-frame prediction mode indicated by the inter-frame prediction mode information. The motion compensation unit 147 supplies the resulting predicted image to the switch 148.

When intra prediction mode information is supplied from the lossless decoding section 132, the switch 148 supplies the predicted image supplied from the intra prediction section 143 to the addition section 135. Meanwhile, when inter prediction mode information is supplied from the lossless decoding section 132, the switch 148 supplies the predicted image supplied from the motion compensation section 147 to the addition section 135.

(Description of Processing of Decoding Device)

FIG22 is a flowchart for describing the image generation process of the decoding device 110 of FIG20 .

In step S111 of FIG. 22 , the reception section 111 of the decoding device 110 receives the encoded stream transmitted from the encoding device 50 of FIG. 13 , and supplies the encoded stream to the extraction section 112 .

In step S112 , the extraction section 112 extracts the encoded data and the parameter set from the encoded stream supplied from the reception section 111 , and supplies the encoded data and the parameter set to the decoding section 113 .

In step S113, the decoding unit 113 performs a decoding process for decoding the encoded data supplied from the extraction unit 112 using the parameter set supplied from the extraction unit 112 as needed, using the HEVC system. The details of the decoding process will be described with reference to FIG. 23 to be described later. Thus, the process is terminated.

FIG23 is a flowchart for describing the details of the decoding process in step S113 of FIG22 .

In step S131 of FIG23 , the accumulation buffer 131 ( FIG21 ) of the decoding section 113 receives the encoded data in frames from the extraction section 112 and accumulates the encoded data. The accumulation buffer 131 supplies the accumulated encoded data to the lossless decoding section 132 as the encoded data of the current image.

In step S132 , the lossless decoding section 132 performs lossless decoding on the encoded data from the accumulation buffer 131 and obtains quantized orthogonal transform coefficients and encoding information. The lossless decoding section 132 supplies the quantized orthogonal transform coefficients to the inverse quantization section 133 .

Furthermore, the lossless decoding section 132 supplies the intra prediction mode information and the like as encoding information to the intra prediction section 143. The lossless decoding section 132 supplies the index of the reference image to the prediction vector generation section 144, supplies the motion vector information to the motion vector generation section 145, and supplies the inter prediction mode information and the index of the reference image to the motion compensation section 147.

Furthermore, the lossless decoding section 132 supplies the intra prediction mode information or the inter prediction mode information as encoding information to the switch 148. The lossless decoding section 132 supplies the offset filter information to the filter 136 as encoding information.

In step S133 , the list creation section 146 performs reference image list creation processing similar to that of FIG. 18 based on the reference image list information contained in the SPS extracted by the extraction section 112 , information arranged in the slice header, and the like.

In step S134 , the inverse quantization section 133 inversely quantizes the quantized orthogonal transform coefficient supplied from the lossless decoding section 132 , and supplies the resultant orthogonal transform coefficient to the inverse orthogonal transform section 134 .

In step S135 , the inverse orthogonal transform section 134 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization section 133 , and supplies the resultant residual information to the addition section 135 .

In step S136, the motion compensation section 147 determines whether or not the inter prediction mode information is supplied from the lossless decoding section 132. In a case where it is determined in step S136 that the inter prediction mode information is supplied, the process proceeds to step S137.

In step S137 , the prediction vector generation section 144 performs a prediction vector list generation process similar to that of FIG. 19 based on the index supplied from the lossless decoding section 132 and the reference image list held in the list creation section 146 .

In step S138, the motion vector generator 145 reads the prediction vector corresponding to the index in the motion vector information from the prediction vector list. The motion vector generator 145 adds the prediction vector to the difference vector in the motion vector information to perform predictive decoding on the current motion vector. The motion vector generator 145 supplies the resulting current motion vector to the motion compensation unit 147.

In step S139, the motion compensation unit 147 performs motion compensation processing on the current block based on the inter-prediction mode information and the reference image index from the lossless decoding unit 132, and the current motion vector from the motion vector generation unit 145. The motion compensation unit 147 supplies the resulting predicted image to the switch 148, and the process proceeds to step S141.

Meanwhile, in a case where determination is made in step S136 that the inter prediction mode information is not supplied, that is, in a case where the intra prediction mode information is supplied to the intra prediction section 143 , the process proceeds to step S140 .

In step S140, the intra prediction unit 143 performs intra prediction processing on the current block in the optimal intra prediction mode indicated by the intra prediction mode information, using the surrounding pixels read from the frame memory 141 via the switch 142. The intra prediction unit 143 supplies the predicted image generated as a result of the intra prediction processing to the addition unit 135 via the switch 148, and the process proceeds to step S141.

In step S141, the addition unit 135 adds the residual information supplied from the inverse orthogonal transform unit 134 and the predicted image supplied from the switch 148 to locally decode the current image. The addition unit 135 supplies the locally decoded current image obtained as a result of the decoding to the frame memory 141. The addition unit 135 also supplies the current image, in which all regions have been decoded, to the filter 136 as a decoded image.

In step S142 , the filter 136 performs a deblocking filter process on the decoded image supplied from the adding section 135 to remove block distortion.

In step S143, the filter 136 performs adaptive offset filtering on the decoded image after the deblocking filtering process for each LCU based on the offset filter information supplied from the lossless decoding section 132. The filter 136 supplies the image after the adaptive offset filtering process to the screen sorting buffer 139 and the frame memory 141.

In step S144, the frame memory 141 stores the current image supplied from the addition section 81 in the cache memory, and stores the decoded image supplied from the filter 136 in the DRAM. Pixels adjacent to the current block in the current image stored in the cache memory are set as peripheral pixels, and are supplied to the intra prediction section 143 via the switch 142. Furthermore, the current image stored in the cache memory or the decoded image stored in the DRAM is set as a reference image and output to the motion compensation section 147 via the switch 142.

In step S145 , the screen sorting buffer 139 stores the decoded images supplied from the filter 136 in frame units, sorts the images in frame units stored in the encoding order into the original display order, and supplies those images to the D/A conversion section 140 .

In step S146, the D/A conversion section 140 performs D/A conversion on the image in units of frames supplied from the screen sorting buffer 139, and outputs the image. The process then returns to step S133 of FIG. 22 and terminates.

As described above, when decoding the current motion vector used in the Intra BC, if the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are different from each other, the decoding device 110 sets the candidate block to be unavailable. Therefore, the motion vector of the candidate block whose actual type of the reference image is different from the actual type of the reference image of the current block is not used to generate the prediction vector.

Therefore, the reference motion vector of the candidate block whose actual type of reference image is different from that of the current block is not used to generate the prediction vector, and thus, an encoded stream with improved encoding efficiency can be decoded.

Furthermore, the decoding device 110 sets the reference image type of the current block, whose actual type is "Intra BC," to LTRP. Therefore, if the actual type of the reference image of the candidate block is STRP, the decoding device 110 sets the candidate block to be unavailable. This makes it possible to decode a coded stream with improved coding efficiency when Intra BC and inter-frame coding are common, without having to significantly change HEVC when Intra BC and inter-frame coding are not common.

However, decoding device 110 registers the image identification information of the current image in a predetermined position in the reference image list of the current block. Therefore, decoding device 110 can determine whether the reference image is the current image based on the index corresponding to the position of the reference image's image identification information in the reference image list. Therefore, when the reference image is the current image, futile access to the DRAM can be prevented.

<Second embodiment>

(Overview of Second Embodiment)

First, an outline of the second embodiment will be described with reference to FIG. 24 to FIG. 27 .

In the second embodiment, the coding efficiency when Intra BC and inter-frame coding are unified is not improved by setting the type of the current image serving as the reference image in Intra BC to LTRP, but is improved by changing the method of generating the prediction vector list when the actual type of the reference image is "Intra BC".

24 to 26 are diagrams for describing a method of generating a prediction vector list in a case where the actual type of a reference image is “Intra BC” in the second embodiment.

It should be noted that DiffPicOrderCnt(X, Y) represents the difference (X-Y) in the POC between image X and image Y. Furthermore, RefPicListX[refIdxLX] and LX[refIdxLX] (X=0 or X=1) respectively represent the reference image identified by the image identification information of the index refIdxLX of the reference image of the current block, among the image identification information registered in the reference image list RefPicListX used for encoding the slice including the current block.

In addition, CurrPic represents the current image, and ColPic represents the image of the temporal candidate block. In addition, xNbAK and yNbAK respectively represent the position in the x direction and the y direction of the spatial candidate block adjacent to the left, upper left, or lower left of the current block (hereinafter referred to as the left candidate block). xNbBK and yNbBK respectively represent the position in the x direction and the y direction of the spatial candidate block adjacent to the upper and upper right of the current block (hereinafter referred to as the upper candidate block).

listCol[refIdxCol] represents a reference image identified by the image identification information of the index refIdxCol of the reference image of the temporal candidate block, among the image identification information registered in the reference image list RefPicListX used for encoding the slice including the temporal candidate block.

As shown in Figure 24, in the second embodiment, in the case where the candidate block to be processed is the left candidate block, when DiffPicOrderCnt(RefPicListX[refIdxLX],CurrPic)=0 and when DiffPicOrderCnt(RefPicListX[refIdxLX],CurrPic) and DiffPicOrderCnt(RefPicListX[RefIdxLX[xNbAK][yNbAK]],CurrPic) are equal to each other, the candidate block to be processed is set to available.

In other words, when the reference image of the current image and the reference image of the left candidate block are both the current image, that is, when the actual type of the reference image of the current image and the actual type of the reference image of the left candidate block are "Intra BC", the left candidate block is set to be available. Then, the candidate of the reference motion vector of the motion vector of the left candidate block is generated as a prediction vector without change.

Similarly, in the second embodiment, as shown in Figure 25, in the case where the candidate block to be processed is the upper candidate block, when DiffPicOrderCnt(RefPicListX[refIdxLX],CurrPic)=0 and when DiffPicOrderCnt(RefPicListX[refIdxLX],CurrPic) and DiffPicOrderCnt(RefPicListX[RefIdxLX[xNbBK][yNbBK]],CurrPic) are equal to each other, the candidate block to be processed is set to available.

In other words, if the actual type of the reference image of the current image and the actual type of the reference image of the upper candidate block are both "Intra BC," the upper candidate block is set to be available. Then, the candidate for the reference motion vector of the motion vector of the upper candidate block is generated as a prediction vector without any changes. This prediction vector generation is performed for each reference image list in which a reference image is registered.

In addition, in the second embodiment, as shown in Figure 26, in the case where the candidate block to be processed is a time direction candidate block, when DiffPicOrderCnt(LX[refIdxLX],CurrPic)=0 and when DiffPicOrderCnt(LX[refIdxLX],CurrPic) and DiffPicOrderCnt(listCol[refIdxCol],ColPic) for colPb are equal to each other, the candidate block to be processed is set to available.

In other words, if the actual type of the reference image of the current image and the actual type of the reference image of the temporal candidate block are both "Intra BC," the temporal candidate block is set to be available. Then, the candidate for the reference motion vector of the motion vector of the temporal candidate block is generated as a prediction vector without any changes.

Therefore, as shown in FIG27 , when the actual type of the reference image of the current image is "Intra BC," and only when the actual type of the reference image of the candidate block is "Intra BC," the motion vector of the candidate block is generated as a prediction vector without change. Meanwhile, in cases other than the above, the candidate block is set to unavailable. It should be noted that the type of a candidate whose reference image actual type is "Intra BC" is set to STRP.

The configurations of the encoding device and decoding device of the second embodiment are similar to those of the encoding device 50 of FIG. 13 and the decoding device 110 of FIG. 20 , except for the reference image list creation processing performed in the list creation section 88 and the list creation section 146, and the prediction vector list generation processing performed in the prediction vector generation section 90 and the prediction vector generation section 144. Therefore, hereinafter, the respective sections of the encoding device 50 of FIG. 13 and the decoding device 110 of FIG. 20 will be used as the respective sections of the encoding device and the decoding device of the second embodiment, and only the reference image list creation processing and the prediction vector generation processing will be described.

(Description of Processing of Encoding Device)

FIG28 is a flowchart for describing a reference image list creation process by the list creation section 88 of the encoding device 50 in the second embodiment.

In step S161 of Fig. 28, the list creation section 88 creates four separate lists similarly to the process of step S71 of Fig. 18. STRP or LTRP is set for the image recognition information of each separate list.

In step S162 , the list creation section 88 creates a current image list in which the image identification information of the current image whose reference image type is set to STRP is registered.

The processing in steps S163 to S166 is similar to the processing in steps S73 to S76 of FIG. 18 , and thus description thereof will be omitted.

It should be noted that, although not shown in the drawings, the reference image list creation process of the list creation section 146 of the decoding device 110 in the second embodiment is also performed similarly to the reference image list creation process of FIG. 28 .

29 is a flowchart for describing details of the prediction vector list generation process of the prediction vector generation section 90 of the encoding device 50 in the second embodiment. The prediction vector list generation process is performed for each reference picture list in which a reference picture is registered.

The processing in steps S181 to S183 of Figure 29 is similar to that in steps S91 to S93 of Figure 19, so its description will be omitted. In the case where it is determined in step S183 that the type of the reference image of the candidate block to be processed is STRP, the processing proceeds to step S184.

In step S184, the prediction vector generation unit 90 determines whether both the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are STRP. Specifically, for example, the prediction vector generation unit 90 determines whether the difference in POC between the reference image of the current block and the current image and the difference in POC between the reference image of the candidate block and the image of the candidate block are both 1 or greater.

When the prediction vector generation unit 90 determines that the difference in POC between the reference image of the current block and the current image and the difference in POC between the reference image of the candidate block and the image of the candidate block are both 1 or greater, the prediction vector generation unit 90 determines that the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are both STRP, and the processing proceeds to step S185.

The processing in steps S185 and S186 is similar to the processing in steps S94 and S95 of FIG. 19 , and thus description thereof will be omitted.

At the same time, when at least one of the differences in POC between the reference image of the current block and the current image and the differences in POC between the reference image of the candidate block and the image of the candidate block is 0, the prediction vector generation unit 90 determines in step S184 that at least one of the actual types of the reference image of the current block and the actual type of the reference image of the candidate block is not STRP.

In step S187, the prediction vector generation unit 90 determines whether both the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are "Intra BC." Specifically, the prediction vector generation unit 90 determines whether the difference in POC between the reference image of the current block and the current image, and the difference in POC between the reference image of the candidate block and the image of the candidate block are zero.

When the prediction vector generation unit 90 determines that the difference in POC between the reference image of the current block and the current image and/or the difference in POC between the reference image of the candidate block and the image of the candidate block is not 0, the prediction vector generation unit 90 determines that either the actual type of the reference image of the current block or the actual type of the reference image of the candidate block is not "Intra BC", and the processing proceeds to step S188.

In other words, in the case where the reference image of the current block is the current image but the image of the candidate block and the reference image of the candidate block are different from each other, and in the case where the image of the candidate block and the reference image of the candidate block are the same but the reference image of the current block is not the current image, the prediction vector generation unit 90 determines that the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are different from each other, and the processing proceeds to step S188.

In step S188 , the prediction vector generation section 90 sets the candidate block to be processed to be unavailable.

In addition, in a case where it is determined in step S182 that the type of the reference image of the current block is not STRP, that is, in a case where the type of the reference image of the current block is LTRP, the process proceeds to step S189.

In step S189, similar to the process of step S183, the prediction vector generation unit 90 determines whether the type of the reference image of the candidate block to be processed is STRP. If it is determined in step S189 that the type of the reference image of the candidate block to be processed is STRP, that is, if the type of the reference image of the current block is LTRP and the type of the reference image of the candidate block is STRP, the process proceeds to step S188. Therefore, the candidate block to be processed is set to unavailable.

Meanwhile, in the case where it is determined in step S189 that the type of the reference image of the candidate block to be processed is not STRP, that is, in the case where both the type of the reference image of the current block and the type of the reference image of the candidate block are LTRP, the process proceeds to step S190. The processes in steps S190 and S191 are similar to those in steps S98 and S99 of FIG. 19, and thus description thereof will be omitted.

At the same time, when the prediction vector generation unit 90 determines that the difference in POC between the reference image of the current block and the current image and the difference in POC between the reference image of the candidate block and the image of the candidate block are both 0, the prediction vector generation unit 90 determines in step S187 that the actual type of the reference image of the current block and the actual type of the reference image of the candidate block are "Intra BC".

Then, the prediction vector generation section 90 advances the process to step S 190. Thus, the candidate block to be processed is set to be available, and a candidate of the reference motion vector of the candidate block to be processed is generated as a prediction vector.

After the processing of step S186, S188 or S191, the processing proceeds to step S 192. The processing of step S192 is similar to the processing of step S100 in FIG19, so the description thereof will be omitted.

It should be noted that, although not shown in the drawings, the prediction vector list generation process of the prediction vector generation section 144 of the decoding device 110 in the second embodiment is also performed similarly to the prediction vector list generation process of FIG. 29 .

<Third embodiment>

(Overview of the Third Embodiment)

In the third embodiment, the image identification information of the current image is registered in the head of the temporary list, not in the reference image list.

30 , after creating two temporary lists RefpicListTemp0 and RefpicListTemp1 based on the individual lists of FIG. 10 , the image identification information of the current image is registered in the head of each temporary list.

Next, the order of the image identification information registered in the RefPicListTemp0 list and the RefPicListTemp1 list is changed as needed (reference record). However, the order of the image identification information of the current image is prohibited from being changed.

Finally, a reference picture list RefPicList0 is created in which the head picture identification information registered in the RefPicListTemp0 list and the number of picture identification information obtained by adding 1 to num_ref_idx_10_active_minus1 (five in the example of FIG. 30 ) are registered in order from the head.

Furthermore, a reference picture list RefPicList1 is created in which the head picture identification information registered in the RefPicListTemp1 list and the number of picture identification information obtained by adding 1 to num_ref_idx_l1_active_minus1 (four in the example of FIG. 30 ) are registered in order from the head.

The configurations of the encoding device and the decoding device of the third embodiment are similar to those of the encoding device 50 of FIG. 13 and the decoding device 110 of FIG. 20 , except for the reference image list creation processing performed in the list creation section 88 and the list creation section 146. Therefore, hereinafter, the respective sections of the encoding device 50 of FIG. 13 and the decoding device 110 of FIG. 20 will be used as the respective sections of the encoding device and the decoding device in the third embodiment, and only the reference image list creation processing will be described.

(Description of Processing of Encoding Device)

FIG31 is a flowchart for describing a reference image list creation process by the list creation section 88 of the encoding device 50 in the third embodiment.

The processing in steps S201 to S203 of FIG. 31 is similar to the processing in steps S71 to S73 of FIG. 18 , and thus description thereof will be omitted.

In step S204 , the list creation unit 88 registers the image identification information of the current image registered in the current image list at the beginning of the temporary list.

In step S205 , the list creation section 88 rearranges the order of the image identification information other than the image identification information of the current image in the temporary list based on ref_pic_list_modification arranged in the slice header.

In step S206, the list creation unit 88 creates a reference picture list based on each temporary list. In other words, the list creation unit 88 creates a reference picture list in which the first picture identification information registered in the temporary list and the number of picture identification information obtained by adding 1 to num_ref_idx_l0_active_minus1 (num_ref_idx_l1_active_minus1) arranged in the slice header are registered in order from the beginning.

It should be noted that, although not shown in the drawings, the reference image list creation process of the list creation section 146 of the decoding device 110 in the third embodiment is also performed similarly to the reference image list creation process of FIG. 31 .

<Fourth embodiment>

(Description of a computer to which the present disclosure is applied)

The series of processes described above can be performed by hardware or software. In the case of performing the series of processes by software, the program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware and, for example, a general-purpose personal computer that can perform various functions by installing various programs therein.

FIG32 is a block diagram showing a hardware configuration example of a computer that executes the above-described series of processes by a program.

In the computer, a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 , and a RAM (Random Access Memory) 203 are connected to one another via a bus 204 .

Furthermore, an input/output interface 205 is connected to the bus 204 . An input section 206 , an output section 207 , a storage section 208 , a communication section 209 , and a drive 210 are connected to the input/output interface 205 .

The input unit 206 is composed of a keyboard, a mouse, a microphone, etc. The output unit 207 is composed of a display, a speaker, etc. The storage unit 208 is composed of a hard disk, a nonvolatile memory, etc. The communication unit 209 is composed of a network interface, etc. The drive 210 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 201 loads a program stored in, for example, the storage section 208 to the RAM 203 via the input/output interface 205 and the bus 204 , and executes the program to perform the above-described series of processing.

The program executed by the computer (CPU 201) can be provided by, for example, recording it in the removable medium 211 such as a package medium. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

In the computer, when the removable medium 211 is mounted to the drive 210, the program can be installed in the storage unit 208 via the input/output interface 205. In addition, the program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. In addition, the program can be pre-installed in the ROM 202 or in the storage unit 208.

It should be noted that the program executed by the computer may be a program that is processed time-sequentially in the order described in this specification, or may be a program that is processed in parallel or at necessary timing such as when a call is executed.

<Fifth embodiment>

(Configuration Example of Television Apparatus)

33 illustrates a schematic configuration of a television device to which the present disclosure is applied. A television device 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, and an external interface unit 909. Furthermore, the television device 900 includes a control unit 910, a user interface unit 911, and the like.

The tuner 902 selects a desired channel from the broadcast signal received by the antenna 901 for demodulation, and outputs the obtained encoded bit stream to the demultiplexer 903 .

The demultiplexer 903 extracts a video or audio packet of a program to be viewed from the encoded bit stream, and outputs the data of the extracted packet to the decoder 904. In addition, the demultiplexer 903 supplies data packets such as an EPG (Electronic Program Guide) to the control section 910. It should be understood that in the case where scrambling is performed, descrambling is performed by the demultiplexer or the like.

The decoder 904 performs a packet decoding process, and outputs the video data generated by the decoding process to the video signal processing section 905 and outputs the audio data generated by the decoding process to the audio signal processing section 907 .

The video signal processing unit 905 performs noise removal and video processing corresponding to user settings on the video signal. The video signal processing unit 905 generates video data for programs to be displayed on the display unit 906, as well as image data processed based on applications supplied via the network. Furthermore, the video signal processing unit 905 generates video data for displaying menu screens for selecting items, and superimposes this video data on the program video data. Based on the generated video data, the video signal processing unit 905 generates a drive signal and drives the display unit 906.

The display section 906 drives a display device (for example, a liquid crystal display element) based on a drive signal from the video signal processing section 905 to display a video of a program or the like.

The audio signal processing section 907 performs predetermined processing such as noise removal on the audio data, performs D/A conversion processing or amplification processing on the processed audio data, and supplies such audio data to the speaker 908 to perform audio output.

The external interface section 909 is an interface for connection with an external device or a network, and transmits and receives data such as video data and audio data.

The user interface section 911 is connected to the control section 910. The user interface section 911 is composed of an operation switch, a remote control signal receiving section, and the like, and supplies an operation signal corresponding to a user operation to the control section 910.

The control unit 910 is composed of a CPU (Central Processing Unit), a memory, and the like. The memory stores programs executed by the CPU, various types of data necessary for the CPU to perform processing, EPG data, data acquired via a network, and the like. The CPU reads and executes the programs stored in the memory at predetermined times, such as when the television device 900 is turned on. The CPU executes the programs to control the various components, causing the television device 900 to operate in accordance with user operations.

It should be noted that in the television apparatus 900 , a bus 912 is provided to connect the tuner 902 , the demultiplexer 903 , the video signal processing section 905 , the audio signal processing section 907 , the external interface section 909 , and the like to the control section 910 .

In the constructed television device, the decoder 904 is provided with the function of the decoding device (decoding method) of the present application. This enables decoding of a coded stream with improved coding efficiency when prediction is performed using correlation within a screen.

<Sixth embodiment>

(Configuration Example of Mobile Phone)

FIG34 illustrates a schematic configuration of a mobile phone to which the present disclosure is applied. A mobile phone 920 includes a communication unit 922, an audio codec 923, a camera unit 926, an image processing unit 927, a multiplexing/demultiplexing unit 928, a recording/reproducing unit 929, a display unit 930, and a control unit 931. These components are connected to each other via a bus 933.

Furthermore, the antenna 921 is connected to the communication section 922, and the speaker 924 and the microphone 925 are connected to the audio codec 923. Furthermore, an operation section 932 is connected to the control section 931.

The mobile phone 920 performs various operations such as transmission and reception of audio signals, transmission and reception of electronic mail or image data, imaging, and data recording in various modes such as a voice communication mode and a data communication mode.

In voice communication mode, the audio signal generated by the microphone 925 is converted into audio data or subjected to data compression in the audio codec 923 and then supplied to the communication unit 922. The communication unit 922 performs modulation processing, frequency conversion processing, etc. on the audio data to generate a transmission signal. In addition, the communication unit 922 supplies the transmission signal to the antenna 921 to transmit it to a base station (not shown in the figure). In addition, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, etc. on the received signal received by the antenna 921, and supplies the obtained audio data to the audio codec 923. The audio codec 923 performs data decompression on the audio data or converts it into an analog audio signal, and outputs the resulting signal to the speaker 924.

Furthermore, when sending mail in data communication mode, the control unit 931 receives character data input through operation of the operation unit 932 and displays the input characters on the display unit 930. Furthermore, the control unit 931 generates mail data based on user instructions and the like from the operation unit 932 and supplies the mail data to the communication unit 922. The communication unit 922 performs modulation processing, frequency conversion processing, and the like on the mail data and transmits the resulting transmission signal from the antenna 921. Furthermore, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like on the received signal received by the antenna 921 to restore the mail data. The mail data is then supplied to the display unit 930 to display the mail content.

It should be understood that the mobile phone 920 also causes the recording/reproducing unit 929 to store the received mail data in a storage medium. The storage medium is any rewritable storage medium. For example, the storage medium is a semiconductor memory such as RAM and built-in flash memory, a hard disk, or a removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, a USB (Universal Serial Bus) memory, and a memory card.

When image data is transmitted in the data communication mode, the image data generated in the camera section 926 is supplied to the image processing section 927. The image processing section 927 performs encoding processing on the image data, and generates encoded data.

The multiplexing/demultiplexing unit 928 multiplexes the coded data generated by the image processing unit 927 and the audio data supplied from the audio codec 923 using a predetermined system, and supplies the resulting data to the communication unit 922. The communication unit 922 performs modulation processing, frequency conversion processing, and other processing on the multiplexed data, and transmits the resulting transmission signal from the antenna 921. Furthermore, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and other processing on the received signal received by the antenna 921 to restore the multiplexed data. The multiplexed data is supplied to the multiplexing/demultiplexing unit 928. The multiplexing/demultiplexing unit 928 demultiplexes the multiplexed data and supplies the coded data to the image processing unit 927 and the audio data to the audio codec 923. The image processing unit 927 performs decoding processing on the coded data to generate image data. The image data is supplied to the display unit 930 to display the received image. The audio codec 923 converts the audio data into an analog audio signal, and supplies the analog audio signal to the speaker 924 to output the received audio.

In the constructed mobile phone device, the image processing unit 927 is provided with the functions of the encoding device and decoding device (encoding method and decoding method) of the present application. This makes it possible to improve encoding efficiency when performing prediction using correlation within a picture. In addition, it is possible to decode a coded stream with improved encoding efficiency when performing prediction using correlation within a picture.

<Seventh embodiment>

(Configuration Example of Recording/Reproducing Device)

FIG35 illustrates a schematic configuration of a recording/reproducing device to which the present disclosure is applied. The recording/reproducing device 940 records, for example, audio and video data of a received broadcast program on a recording medium and provides the recorded data to the user at a time corresponding to a user instruction. Furthermore, the recording/reproducing device 940 can also obtain, for example, audio and video data from another device and record that data on a recording medium. Furthermore, the recording/reproducing device 940 decodes and outputs the audio and video data recorded on the recording medium so that monitoring equipment, etc., can perform image display or audio output.

The recording/reproducing device 940 includes a tuner 941, an external interface section 942, an encoder 943, an HDD (hard disk drive) section 944, a disk drive 945, a selector 946, a decoder 947, an OSD (on screen display) section 948, a control section 949 and a user interface section 950.

The tuner 941 selects a desired channel from broadcast wave signals received by an antenna (not shown in the figure), and outputs an encoded bit stream obtained by demodulating the received signal of the desired channel to the selector 946 .

The external interface unit 942 is configured from at least one of an IEEE1394 interface, a network interface, a USB interface, a flash memory interface, etc. The external interface unit 942 is an interface for connecting to an external device, a network, a memory card, etc., and receives data such as video data or audio data to be recorded.

When the video data or the audio data supplied from the external interface section 942 is not encoded, the encoder 943 performs encoding by a predetermined system and outputs an encoded bit stream to the selector 946 .

The HDD section 944 records content data such as video and audio, various programs, and other data in a built-in hard disk, and reads those data from the hard disk at the time of reproduction or the like.

The disk drive 945 records and reproduces signals on a loaded optical disk, such as a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW, etc.) or a Blu-ray (registered trademark) disk.

When recording video or audio, the selector 946 selects an arbitrary encoded bit stream from the tuner 941 or the encoder 943, and supplies the encoded bit stream to either the HDD unit 944 or the disk drive 945. Furthermore, when reproducing video or audio, the selector 946 supplies the encoded bit stream output from the HDD unit 944 or the disk drive 945 to the decoder 947.

The decoder 947 performs a decoding process on the encoded bit stream. The decoder 947 supplies the video data generated by the decoding process to the OSD section 948. Furthermore, the decoder 947 outputs the audio data generated by the decoding process.

The OSD section 948 generates video data for displaying a menu screen to select an item or the like, and superimposes such a video signal on the video data output from the decoder 947 to output it.

The user interface section 950 is connected to the control section 949. The user interface section 950 is configured by an operation switch, a remote control signal receiving section, and the like, and supplies an operation signal corresponding to a user operation to the control section 949.

The control unit 949 is composed of a CPU, a memory, and the like. The memory stores programs executed by the CPU and various types of data necessary for the CPU to perform processing. The CPU reads and executes the programs stored in the memory at predetermined times, such as when the recording/reproducing device 940 is activated. The CPU executes the programs to control various components, causing the recording/reproducing device 940 to operate in accordance with user operations.

In the recording/reproducing device thus constructed, the encoder 943 is provided with the function of the encoding device (encoding method) of the present application, and the decoder 947 is provided with the function of the decoding device (decoding method) of the present application. This makes it possible to improve encoding efficiency when performing prediction using correlation within a picture. In addition, it is possible to decode an encoded stream having improved encoding efficiency when performing prediction using correlation within a picture.

<Eighth Embodiment>

(Configuration Example of Imaging Apparatus)

36 illustrates a schematic configuration of an imaging device to which the present disclosure is applied. The imaging device 960 images a subject and then displays the image of the subject on a display section or records such image data on a recording medium.

The imaging device 960 includes an optical block 961, an imaging section 962, a camera signal processing section 963, an image data processing section 964, a display section 965, an external interface section 966, a storage section 967, a media driver 968, an OSD section 969, and a control section 970. Furthermore, a user interface section 971 is connected to the control section 970. Furthermore, the image data processing section 964, the external interface section 966, the storage section 967, the media driver 968, the OSD section 969, the control section 970, and the like are connected to one another via a bus 972.

The optical block 961 is composed of a focusing lens, an aperture mechanism, etc. The optical block 961 forms an optical image of a subject on an imaging surface of the imaging section 962. The imaging section 962 is composed of a CCD or CMOS image sensor, generates an electrical signal corresponding to the optical image through photoelectric conversion, and supplies the electrical signal to the camera signal processing section 963.

The camera signal processing section 963 performs various types of camera signal processing such as knee correction, gamma correction, and color correction on the electrical signal supplied from the imaging section 962. The camera signal processing section 963 supplies the image data after undergoing the camera signal processing to the image data processing section 964.

The image data processing section 964 performs encoding processing on the image data supplied from the camera signal processing section 963. The image data processing section 964 supplies the encoded data generated by the encoding processing to the external interface section 966 or the media drive 968. The image data processing section 964 also performs decoding processing on the encoded data supplied from the external interface section 966 or the media drive 968. The image data processing section 964 supplies the image data generated by the decoding processing to the display section 965. The image data processing section 964 also performs processing to supply the image data supplied from the camera signal processing section 963 to the display section 965, or superimposes display data acquired from the OSD section 969 on the image data and supplies such display data to the display section 965.

The OSD portion 969 generates display data of a menu screen composed of symbols, characters, graphics, icons, and the like, and outputs the display data to the image data processing portion 964 .

The external interface unit 966 is composed of, for example, USB input and output terminals, and is connected to a printer in the case of printing an image. In addition, a drive is connected to the external interface unit 966 as needed, and removable media such as disks and optical disks are appropriately installed so that computer programs read therefrom can be installed as needed. In addition, the external interface unit 966 includes a network interface connected to a predetermined network such as a LAN (local area network) and the Internet. The control unit 970 can, for example, read coded data from the media drive 968 according to an instruction from the user interface unit 971, and can supply the coded data from the external interface unit 966 to another device connected via the network. In addition, the control unit 970 can obtain coded data or image data supplied from another device via the network via the external interface unit 966, and supply the data to the image data processing unit 964.

As the recording medium driven by the media drive 968, for example, any readable and writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory is used. In addition, the recording medium may be any type of recording medium such as a removable medium, and may be a magnetic tape device, an optical disk, or a memory card. Of course, the recording medium may be a contactless IC (Integrated Circuit) card or the like.

Furthermore, the media drive 968 and the recording medium may be integrated and configured from a non-portable storage medium such as a built-in hard disk drive and an SSD (Solid State Drive).

The control unit 970 is composed of a CPU. The storage unit 967 stores programs executed by the control unit 970, various types of data necessary for the control unit 970 to perform processing, and the like. The control unit 970 reads and executes the programs stored in the storage unit 967 at predetermined times, such as when the imaging device 960 is activated. The control unit 970 executes the programs to control various components, causing the imaging device 960 to operate in accordance with user operations.

In the constructed imaging device, the image data processing unit 964 is provided with the functions of the encoding device and decoding device (encoding method and decoding method) of the present application. This makes it possible to improve encoding efficiency when performing prediction using correlation within a picture. In addition, it is possible to decode a coded stream with improved encoding efficiency when performing prediction using correlation within a picture.

Ninth embodiment

(Other Examples of Embodiments)

Examples of devices, systems, etc. to which the present disclosure is applied have been described above, but the present disclosure is not limited thereto and may be implemented as any other configuration to be installed in devices constituting the above-mentioned devices or systems, for example, a processor as a system LSI (large-scale integration), a module using multiple processors, a unit using multiple modules, a set obtained by further adding additional functions to a unit, etc. (i.e., a configuration of a portion of a device).

(Example of video kit configuration)

An example of a case where the present disclosure is implemented as a video suite will be described with reference to Fig. 37. Fig. 37 shows an example of a schematic configuration of a video suite to which the present disclosure is applied.

In recent years, the multifunctionality of electronic devices has been progressing, and in the development and manufacture of electronic devices, when a part of their structure is implemented through sales, provision, etc., it is frequently found that not only is the part implemented as a structure having one function, but it is also frequently found that the part is implemented as a set having multiple functions by combining multiple structures having related functions.

The video set 1300 shown in FIG. 37 has such a multifunctional structure and is a combination of a device having image encoding and decoding (either or both) functions and a device having another function associated with the functions.

As shown in Figure 37, the video suite 1300 includes: a module group, which includes a video module 1311, an external memory 1312, a power management module 1313, a front-end module 1314, etc.; and devices with associated functions such as a connector 1321, a camera 1322 and a sensor 1323.

A module is assumed to be a component in which several related component-like functions are integrated to provide coherent functionality. The specific physical configuration is arbitrary, but it is assumed to be a configuration obtained by, for example, arranging multiple processors with corresponding functions, electronic circuit elements such as resistors and capacitors, and other devices on a wiring substrate for integration. Furthermore, it is also assumed that a module can be combined with another module, a processor, etc. to form a new module.

In the case of the example of FIG. 37 , the video module 1311 is a combination of configurations having an image processing function, and includes an application processor, a video processor, a broadband modem 1333 , and an RF (Radio Frequency) module 1334 .

A processor is a component in which a structure having a predetermined function is integrated into a semiconductor chip by SoC (System on Chip), and may be referred to as, for example, System LSI (Large Scale Integration). Such a structure having a predetermined function may be a logic circuit (hardware structure), a CPU, ROM, RAM, etc., and a program executed by using the above structure (software structure), or may be a combination of the above structures. For example, a processor may include a logic circuit, a CPU, ROM, RAM, etc., and may implement a part of the function by the logic circuit (hardware structure) and implement other functions by a program executed by the CPU (software structure).

The application processor 1331 in FIG37 is a processor that executes an application related to image processing. The application executed by the application processor 1331 can not only perform computing processing, but also control the internal and external components of the video module 1311, such as the video processor 1332, as needed to achieve predetermined functions.

The video processor 1332 is a processor having functions of image encoding/decoding (one or both).

The broadband modem 1333 is a processor (or module) that processes wired or wireless (or both) broadband communications via a broadband connection, such as the Internet and a public telephone network. For example, the broadband modem 1333 performs digital modulation on transmitted data (digital signals) to convert them into analog signals, or demodulates received analog signals to convert them into data (digital signals). For example, the broadband modem 1333 can perform digital modulation/demodulation on arbitrary information, such as image data processed by the video processor 1332, encoded image data streams, application programs, and configuration data.

The RF module 1334 is a module that performs frequency conversion, modulation and demodulation, amplification, filtering, and the like on RF (Radio Frequency) signals transmitted and received via the antenna. For example, the RF module 1334 performs frequency conversion, etc. on the baseband signal generated by the broadband modem 1333, and generates an RF signal. In addition, for example, the RF module 1334 performs frequency conversion, etc. on the RF signal received via the front-end module 1314, and generates a baseband signal.

It should be noted that, as indicated by a dotted line 1341 in FIG. 37 , the application processor 1331 and the video processor 1332 may be integrated and constituted as one processor.

The external memory 1312 is a module provided outside the video module 1311 and includes a storage device used by the video module 1311. The storage device of the external memory 1312 can be implemented by any physical structure. Generally, since the storage device is frequently used to store a large amount of data such as image data in units of frames, the storage device is preferably implemented by, for example, a relatively inexpensive and large-capacity semiconductor memory such as DRAM (Dynamic Random Access Memory).

The power management module 1313 manages and controls the power supply to the video module 1311 (the configuration within the video module 1311 ).

The front-end module 1314 is a module that provides a front-end function (circuitry at the transmitting and receiving ends on the antenna side) to the RF module 1334. As shown in FIG37 , the front-end module 1314 includes, for example, an antenna portion 1351, a filter 1352, and an amplifier portion 1353.

Antenna section 1351 includes an antenna and its peripheral components for transmitting and receiving radio signals. Antenna section 1351 transmits the signal supplied from amplifier section 1353 as a radio signal and supplies the received radio signal to filter 1352 as an electrical signal (RF signal). Filter 1352 performs filtering and other processing on the RF signal received via antenna section 1351 and supplies the processed RF signal to RF module 1334. Amplifier section 1353 amplifies the RF signal supplied from RF module 1334 and supplies the resulting signal to antenna section 1351.

The connector 1321 is a module having an external connection function. The physical structure of the connector 1321 is arbitrary. For example, the connector 1321 includes a structure having a communication function other than a communication standard corresponding to the broadband modem 1333, an external input/output terminal, etc.

For example, the connector 1321 may include a module having a communication function according to wireless communication standards such as Bluetooth (registered trademark), IEEE802.11 (e.g., Wi-Fi (Wireless Fidelity, registered trademark)), NFC (Near Field Communication), and IrDA (Infrared Data Association), an antenna for transmitting and receiving signals according to such standards, etc. In addition, for example, the connector 1321 may include a module having a communication function according to wired communication standards such as USB (Universal Serial Bus) and HDMI (registered trademark) (High-Definition Multimedia Interface), and terminals according to such standards. In addition, for example, the connector 1321 may have additional data (signal) transmission functions, such as module input/output terminals.

It should be noted that the connection member 1321 may include a device that serves as a data (signal) transmission destination. For example, the connection member 1321 may include a drive that reads and writes data to recording media such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memories (including not only drives for removable media but also hard disks, SSDs (solid-state drives), and NAS (network attached storage)). Furthermore, the connection member 1321 may include an image or audio output device (monitor, speaker, etc.).

The camera 1322 is a module having a function of imaging a subject and acquiring image data of the subject. Image data obtained by imaging by the camera 1322 is supplied to the video processor 1332 and encoded, for example.

The sensor 1323 is a module having the functions of any sensor, such as an audio sensor, an ultrasonic sensor, an optical sensor, an illumination sensor, an infrared sensor, an image sensor, a rotation sensor, an angle sensor, an angular velocity sensor, a speed sensor, an acceleration sensor, a tilt sensor, a magnetic recognition sensor, a vibration sensor, and a temperature sensor. Data detected by the sensor 1323 is supplied to the application processor 1331, for example, and used by an application or the like.

The configuration described above as a module may be implemented as a processor, and conversely, the configuration described as a processor may be implemented as a module.

In the video suite 1300 constructed as described above, the present disclosure can be applied to a video processor 1332 to be described later. Therefore, the video suite 1300 can be implemented as a suite to which the present disclosure is applied.

(Configuration Example of Video Processor)

FIG38 shows an example of a schematic configuration of the video processor 1332 ( FIG37 ) to which the present disclosure is applied.

In the case of the example of FIG. 38 , the video processor 1332 has a function of receiving input of video signals and audio signals and encoding those signals through a predetermined system, and a function of decoding the encoded video signals and audio signals and reproducing and outputting the video signals and audio signals.

As shown in FIG38 , the video processor 1332 includes a video input processing unit 1401, a first image scaling unit 1402, a second image scaling unit 1403, a video output processing unit 1404, a frame memory 1405, and a memory control unit 1406. Furthermore, the video processor 1332 includes an encoding/decoding engine 1407, video ES (elementary stream) buffers 1408A and 1408B, and audio ES buffers 1409A and 1409B. Furthermore, the video processor 1332 includes an audio encoder 1410, an audio decoder 1411, a multiplexing unit (MUX (Multiplexer) 1412), a demultiplexing unit (DMUX (Demultiplexer)) 1413, and a stream buffer 1414.

The video input processing unit 1401 receives a video signal input from, for example, the connector 1321 ( FIG. 37 ), and converts the video signal into digital image data. The first image scaling unit 1402 performs format conversion, image scaling, and other processing on the image data. The second image scaling unit 1403 performs image scaling processing on the image data in accordance with the format of the destination for output via the video output processing unit 1404, or performs format conversion, image scaling, and other processing similar to that of the first image scaling unit 1402. The video output processing unit 1404 performs format conversion, conversion into an analog signal, and other processing on the image data, and outputs the data as a reproduced video signal to, for example, the connector 1321 ( FIG. 37 ).

The frame memory 1405 is a memory for image data shared with the video input processing unit 1401, the first image scaling unit 1402, the second image scaling unit 1403, the video output processing unit 1404, and the encoding/decoding engine 1407. The frame memory 1405 is implemented as a semiconductor memory such as a DRAM.

In response to a synchronization signal from the encoding/decoding engine 1407, the memory control unit 1406 controls access to and reading from the frame memory 1405 according to the schedule for accessing the frame memory 1405 written to the access management table 1406A. The access management table 1406A is updated by the memory control unit 1406 based on the processing executed by the encoding/decoding engine 1407, the first image scaling unit 1402, the second image scaling unit 1403, and the like.

The encoding/decoding engine 1407 encodes the image data and decodes the video stream, which contains the encoded image data. For example, the encoding/decoding engine 1407 encodes the image data read from the frame memory 1405 and then writes the image data as a video stream to the video ES buffer 1408A. Furthermore, for example, the encoding/decoding engine 1407 then reads the video stream from the video ES buffer 1408B for decoding and then writes the decoded video stream as image data to the frame memory 1405. During encoding and decoding, the encoding/decoding engine 1407 uses the frame memory 1405 as a work area. Furthermore, for example, when starting macroblock-based processing, the encoding/decoding engine 1407 outputs a synchronization signal to the memory control unit 1406.

The video ES buffer 1408A buffers the video stream generated by the encoding/decoding engine 1407 and supplies the video stream to the multiplexing section (MUX) 1412. The video ES buffer 1408B buffers the video stream supplied from the demultiplexing section (DMUX) 1413 and supplies the video stream to the encoding/decoding engine 1407.

The audio ES buffer 1409A buffers the audio stream generated by the audio encoder 1410 and supplies the audio stream to the multiplexing section (MUX) 1412. The audio ES buffer 1409B buffers the audio stream supplied from the demultiplexing section (DMUX) 1413 and supplies the audio stream to the audio decoder 1411.

The audio encoder 1410 performs, for example, digital conversion on the audio signal input from, for example, the connector 1321 ( FIG. 37 ), and encodes the resulting signal using a predetermined system such as the MPEG audio system and the AC3 (Audio Coding 3) system. The audio encoder 1410 then writes the audio stream, which is the data in which the audio signal is encoded, into the audio ES buffer 1409A. The audio decoder 1411 decodes the audio stream supplied from the audio ES buffer 1409B, converts it into, for example, an analog signal, and supplies the analog signal as a reproduced audio signal to, for example, the connector 1321 ( FIG. 37 ).

The multiplexing unit (MUX) 1412 multiplexes the video stream and the audio stream. The multiplexing method (i.e., the format of the bit stream generated by multiplexing) is arbitrary. In addition, during multiplexing, the multiplexing unit (MUX) 1412 can also add predetermined header information, etc. to the bit stream. In other words, the multiplexing unit (MUX) 1412 converts the format of the stream by multiplexing. For example, the multiplexing unit (MUX) 1412 multiplexes the video signal and the audio signal to convert those streams into a transport stream, which is a bit stream having a format for transmission. In addition, for example, the multiplexing unit (MUX) 1412 multiplexes the video signal and the audio signal to convert those streams into data (file data) having a file format for recording.

The demultiplexing unit (DMUX) 1413 demultiplexes the bit stream in which the video stream and the audio stream are multiplexed by a method corresponding to the multiplexing of the multiplexing unit (MUX) 1412. In other words, the demultiplexing unit (DMUX) 1413 extracts the video stream and the audio stream from the bit stream read from the stream buffer 1414 (separates the video stream and the audio stream from each other). In other words, the demultiplexing unit (DMUX) 1413 can convert the format of the stream by demultiplexing (the inverse conversion of the conversion of the multiplexing unit (MUX) 1412). For example, the demultiplexing unit (DMUX) 1413 can obtain a transport stream supplied from, for example, the connector 1321 or the broadband modem 1333 (each of which is shown in FIG. 37) via the stream buffer 1414, and demultiplex the transport stream to convert the transport stream into a video stream and an audio stream. In addition, for example, the demultiplexing unit (DMUX) 1413 can obtain file data read from various recording media through, for example, the connector 1321 (Figure 37) and the stream buffer 1414, and demultiplex the file data to convert the file data into video streams and audio streams.

The stream buffer 1414 buffers the bit stream. For example, the stream buffer 1414 buffers the transport stream supplied from the multiplexing unit (MUX) 1412, and supplies the transport stream to, for example, the connection 1321 or the broadband modem 1333 (each of which is shown in FIG. 37 ) at a predetermined timing or based on a request from the outside world.

In addition, for example, the stream buffer 1414 buffers the file data supplied from the multiplexing unit (MUX) 1412, supplies the file data to, for example, the connector 1321 (Figure 37) at a predetermined time or based on a request from the outside world, and records the file data on various recording media.

In addition, the stream buffer 1414 buffers the transmission stream obtained via, for example, the connection member 1321 or the broadband modem 1333 (each of which is shown in Figure 37), and supplies the transmission stream to the demultiplexing unit (DMUX) 1413 at a predetermined time or based on a request from the outside world, etc.

Furthermore, the stream buffer 1414 buffers file data read from various recording media in, for example, the connector 1321 ( FIG. 37 ), and supplies the file data to the demultiplexing section (DMUX) 1413 at a predetermined timing or based on a request from the outside or the like.

Next, an example of the operation of the video processor 1332 having such a configuration will be described. For example, in the video input processing section 1401, a video signal input from the connector 1321 ( FIG. 37 ) or the like to the video processor 1332 is converted into digital image data of a predetermined system, such as a 4:2:2 Y/Cb/Cr system, and then written into the frame memory 1405. The digital image data is read by the first image scaling section 1402 or the second image scaling section 1403, subjected to format conversion to a predetermined system, such as a 4:2:0 Y/Cb/Cr system, and scaling processing, and then written again into the frame memory 1405. The image data is encoded by the encoding/decoding engine 1407 and written as a video stream into the video ES buffer 1408A.

Furthermore, the audio signal input from the connector 1321 ( FIG. 37 ) or the like to the video processor 1332 is encoded by the audio encoder 1410 and written in the audio ES buffer 1409A as an audio stream.

The video stream of the video ES buffer 1408A and the audio stream of the audio ES buffer 1409A are read and multiplexed by the multiplexing unit (MUX) 1412, and are converted into a transport stream, file data, etc. The transport stream generated by the multiplexing unit (MUX) 1412 is buffered by the stream buffer 1414 and then output to the external network via, for example, the connection 1321 or the broadband modem 1333 (each of which is shown in Figure 37). In addition, the file data generated by the multiplexing unit (MUX) 1412 is buffered by the stream buffer 1414 and then output to the connection 1321 (Figure 37) and recorded on various recording media.

Furthermore, a transport stream input to the video processor 1332 from an external network via, for example, the connection 1321 or the broadband modem 1333 (each of which is shown in FIG. 37 ) is buffered in the stream buffer 1414 and then demultiplexed by the demultiplexing section (DMUX) 1413. Furthermore, file data read from various recording media in, for example, the connection 1321 ( FIG. 37 ) and input to the video processor 1332 is buffered in the stream buffer 1414 and then demultiplexed by the demultiplexing section (DMUX) 1413. In other words, the transport stream or file data input to the video processor 1332 is separated into a video stream and an audio stream by the demultiplexing section (DMUX) 1413.

The audio stream is supplied to the audio decoder 1411 via the audio ES buffer 1409B and decoded to reproduce the audio signal. Furthermore, the video stream is written to the video ES buffer 1408B, then read by the encoding/decoding engine 1407, decoded, and written to the frame memory 1405. The decoded image data undergoes scaling processing by the second image scaling unit 1403 and is written to the frame memory 1405. The decoded image data is then read by the video output processing unit 1404, subjected to format conversion to a predetermined system, such as the 4:2:2 Y/Cb/Cr system, and converted into an analog signal to reproduce and output the video signal.

In the case where the present disclosure is applied to the video processor 1332 configured as described above, it is only necessary to apply the present disclosure according to each of the above-described embodiments to the encoding/decoding engine 1407. In other words, for example, the encoding/decoding engine 1407 only needs to have the functions of the encoding device or decoding device according to the first to third embodiments. This enables the video processor 1332 to obtain effects similar to those described above with reference to FIG. 1 to FIG. 31.

It should be noted that in the encoding/decoding engine 1407, the present disclosure (i.e., the functions of the encoding device or decoding device according to each of the above-mentioned embodiments) can be implemented by hardware such as a logic circuit, can be implemented by software such as an embedded program, or can be implemented by both.

(Another Configuration Example of a Video Processor)

Fig. 39 shows another example of a schematic configuration of the video processor 1332 (Fig. 37) to which the present disclosure is applied. In the example of Fig. 39, the video processor 1332 has a function of encoding/decoding video data by a predetermined system.

39 , the video processor 1332 includes a control unit 1511, a display interface 1512, a display engine 1513, an image processing engine 1514, and an internal memory 1515. Furthermore, the video processor 1332 includes a codec engine 1516, a memory interface 1517, a multiplexing/demultiplexing unit (MUX DMUX) 1518, a network interface 1519, and a video interface 1520.

The control unit 1511 controls the operations of corresponding processing units within the video processor 1332 , such as the display interface 1512 , the display engine 1513 , the image processing engine 1514 , and the codec engine 1516 .

As shown in FIG39 , the control unit 1511 includes, for example, a main CPU 1531, a sub-CPU 1532, and a system controller 1533. The main CPU 1531 executes a program for controlling the operation of the corresponding processing unit in the video processor 1332. The main CPU 1531 generates a control signal based on the program and supplies the control signal to the processing unit (i.e., controls the operation of the corresponding processing unit). The sub-CPU 1532 plays an auxiliary role to the main CPU 1531. For example, the sub-CPU 1532 executes a sub-process, a subroutine, etc. of the program executed by the main CPU 1531 and the like. The system controller 1533 controls the operation of the main CPU 1531 and the sub-CPU 1532, such as specifying the program executed by the main CPU 1531 and the sub-CPU 1532.

The display interface 1512 outputs the image data to, for example, the connector 1321 ( FIG. 37 ) under the control of the control unit 1511. For example, the display interface 1512 converts the image data of digital data into an analog signal and outputs the analog signal as a reproduced video signal or the image data of digital data without changing it to a monitoring device or the like of the connector 1321 ( FIG. 37 ).

The display engine 1513 performs various types of conversion processing such as format conversion, size conversion, and color gamut conversion on image data under the control of the control section 1511 to match it with the hardware specifications of a monitor device or the like that displays the image data.

The image processing engine 1514 performs predetermined image processing such as filtering on the image data under the control of the control unit 1511 for improving image quality, for example.

The internal memory 1515 is provided within the video processor 1332 and is shared with the display engine 1513, the image processing engine 1514, and the codec engine 1516. The internal memory 1515 is used, for example, to transfer data between the display engine 1513, the image processing engine 1514, and the codec engine 1516. For example, the internal memory 1515 stores data supplied from the display engine 1513, the image processing engine 1514, or the codec engine 1516, and supplies data to the display engine 1513, the image processing engine 1514, or the codec engine 1516 as needed (e.g., in response to a request). The internal memory 1515 can be implemented by any storage device. Generally, since the internal memory 1515 is frequently used to store small amounts of data, such as image data and parameters in units of blocks, it is desirable to implement the internal memory 1515 by a semiconductor memory device, such as SRAM (static random access memory), that has a relatively small capacity (compared to, for example, the external memory 1312) but high-speed response.

The codec engine 1516 performs encoding or decoding processing on the image data. The encoding/decoding system corresponding to the codec engine 1516 is arbitrary, and the number of such systems may be one or more. For example, the codec engine 1516 may have codec functions of multiple encoding/decoding systems and perform encoding of the image data or decoding of the encoded data using a function selected from the above functions.

In the example shown in Figure 39, the codec engine 1516 includes, for example, MPEG-2 Video 1541, AVC/H.2641542, HEVC/H.2651543, HEVC/H.265 (Scalable) 1544, HEVC/H.265 (Multi-view) 1545 and MPEG-DASH 1551 as functional blocks for processing regarding the codec.

MPEG-2 Video 1541 is a functional block for encoding and decoding image data using the MPEG-2 system. AVC/H.264 1542 is a functional block for encoding and decoding image data using the AVC system. HEVC/H.265 1543 is a functional block for encoding and decoding image data using HEVC. HEVC/H.265 (Scalable) 1544 is a functional block for scalable encoding and scalable decoding of image data using HEVC. HEVC/H.265 (Multiview) 1545 is a functional block for multi-view encoding and multi-view decoding of image data using HEVC.

MPEG-DASH 1551 is a functional block for transmitting and receiving image data via the MPEG-DASH (MPEG-Dynamic Adaptive Streaming over HTTP) system. MPEG-DASH is a video streaming technology that uses HTTP (Hypertext Transfer Protocol), and one of its features is that it selects and transmits appropriate encoded data in units of segments from pre-prepared encoded data of different resolutions, etc. MPEG-DASH 1551 performs stream generation, stream transmission and control, etc. according to the standard, and uses the above-mentioned MPEG-2 Video 1541 to HEVC/H.265 (Multiview) 1545 for encoding and decoding image data.

The memory interface 1517 is an interface for the external memory 1312. Data supplied from the image processing engine 1514 or the codec engine 1516 is supplied to the external memory 1312 via the memory interface 1517. In addition, data read from the external memory 1312 is supplied to the video processor 1332 (the image processing engine 1514 or the codec engine 1516) via the memory interface 1517.

The multiplexing/demultiplexing unit (MUX DMUX) 1518 multiplexes or demultiplexes various types of data about images, such as the bit stream of encoded data, image data, and video signals. The multiplexing/demultiplexing method is arbitrary. For example, during multiplexing, the multiplexing/demultiplexing unit (MUX DMUX) 1518 can not only integrate several pieces of data into one piece of data, but also add predetermined header information, etc. to the data. In addition, during demultiplexing, the multiplexing/demultiplexing unit (MUX DMUX) 1518 can not only divide one piece of data into several pieces of data, but also add predetermined header information, etc. to the divided data. In other words, the multiplexing/demultiplexing unit (MUX DMUX) 1518 can convert the data format by multiplexing/demultiplexing. For example, the multiplexing/demultiplexing section (MUX DMUX) 1518 can convert the bit stream into a transport stream or data (file data) having a file format for recording by multiplexing the bit stream. Of course, the reverse conversion can also be performed by demultiplexing.

The network interface 1519 is an interface for, for example, the broadband modem 1333, the connector 1321 (each of which is shown in FIG37), etc. The video interface 1520 is an interface for, for example, the connector 1321, the camera 1322 (each of which is shown in FIG37), etc.

Next, an example of the operation of the video processor 1332 described above will be described. For example, when a transport stream is received from an external network via the connection 1321, broadband modem 1333 (each of which is shown in FIG37 ), etc., the transport stream is supplied to the multiplexing/demultiplexing unit (MUX DMUX) 1518 via the network interface 1519, demultiplexed, and then decoded by the codec engine 1516. The image data obtained by decoding by the codec engine 1516 undergoes predetermined image processing by the image processing engine 1514, undergoes predetermined conversion by the display engine 1513, and is supplied to, for example, the connection 1321 ( FIG37 ) via the display interface 1512, so that its image is displayed on a monitor. In addition, for example, the image data obtained by decoding by the codec engine 1516 is encoded again by the codec engine 1516, multiplexed and converted into file data by the multiplexing/demultiplexing unit (MUX DMUX) 1518, output to, for example, the connector 1321 (Figure 37) via the video interface 1520, and recorded on various recording media.

Furthermore, for example, the file data of the coded data is supplied to the multiplexing/demultiplexing section (MUX DMUX) 1518 via the video interface 1520, and the file data is demultiplexed and then decoded by the codec engine 1516. The file data of the coded data in which the image data is coded is read from a recording medium (not shown) via the connector 1321 (FIG. 37) or the like. The image data obtained by decoding by the codec engine 1516 undergoes predetermined image processing by the image processing engine 1514, undergoes predetermined conversion by the display engine 1513, and is supplied to the connector 1321 (FIG. 37) or the like via the display interface 1512, so that the image thereof is displayed on a monitor. In addition, for example, the image data obtained by decoding by the codec engine 1516 is encoded again by the codec engine 1516, multiplexed and converted into a transport stream by the multiplexing/demultiplexing unit (MUX DMUX) 1518, and supplied to, for example, the connector 1321, the broadband modem 1333 (each of which is shown in FIG37 ), etc. via the network interface 1519, and transmitted to another device not shown in the figure.

It should be noted that giving and receiving image data or other data between the processing sections of the video processor 1332 is performed by using, for example, the internal memory 1515 or the external memory 1312. Furthermore, for example, the power management module 1313 controls power supply to the control section 1511.

When the present disclosure is applied to the video processor 1332 configured as described above, it is only necessary to apply the present disclosure according to each of the above-described embodiments to the codec engine 1516. In other words, for example, the codec engine 1516 only needs to include a functional block that implements the encoding device or decoding device according to the first to third embodiments. With the configured codec engine 1516, the video processor 1332 can achieve effects similar to those described above with reference to FIG. 1 to FIG. 31.

It should be noted that in the codec engine 1516, the present disclosure (i.e., the functions of the encoding device or decoding device according to each of the above-mentioned embodiments) can be implemented by hardware such as a logic circuit, can be implemented by software such as an embedded program, or can be implemented by both.

Two examples of the configuration of the video processor 1332 have been described above, but the configuration of the video processor 1332 is arbitrary and may be other than the two examples described above. In addition, the video processor 1332 may be configured as a single semiconductor chip or may be configured as multiple semiconductor chips. For example, the video processor 1332 may be a three-dimensional stacked LSI in which multiple semiconductors are stacked. In addition, the video processor 1332 may be implemented using multiple LSIs.

(Example of application to a device)

The video suite 1300 can be incorporated into various devices that process image data. For example, the video suite 1300 can be incorporated into the television device 900 ( FIG. 33 ), the mobile phone 920 ( FIG. 34 ), the recording/reproducing device 940 ( FIG. 35 ), the imaging device 960 ( FIG. 36 ), and the like. By incorporating the video suite 1300 into a device, the device can achieve effects similar to those described above with reference to FIG. 1 to FIG. 31 .

It should be understood that if each component of the video suite 1300 includes a video processor 1332, that component can be implemented as a component to which the present disclosure applies. For example, only the video processor 1332 can be implemented as a video processor to which the present disclosure applies. Furthermore, for example, as described above, the processor indicated by the dotted line 1341, the video module 1311, and the like can be implemented as processors, modules, and the like to which the present disclosure applies. Furthermore, for example, the video module 1311, the external memory 1312, the power management module 1313, and the front-end module 1314 can be combined to form a video unit 1361 to which the present disclosure applies. In any of these configurations, effects similar to those described above with reference to FIG. 1 to FIG. 31 can be achieved.

In other words, similar to the case of video suite 1300, any configuration including video processor 1332 can be incorporated into various devices that process image data. For example, video processor 1332, the processor indicated by dotted line 1341, video module 1311, or video unit 1361 can be incorporated into television device 900 (FIG. 33), mobile phone 920 (FIG. 34), recording/reproducing device 940 (FIG. 35), imaging device 960 (FIG. 36), and the like. Similar to the case of video suite 1300, by incorporating and applying any configuration of the present disclosure, such devices can achieve effects similar to those described above with reference to FIG. 1 to FIG. 31.

It should be understood that the following example has been described in this specification: various types of information are multiplexed into encoded data and transmitted from the encoding side to the decoding side. However, the technology for transmitting those information is not limited to such an example. For example, those information can be transmitted or recorded as separate data associated with the encoded data without multiplexing them into the encoded data. Here, the term "associated" means that the image included in the bit stream (which can be a part of the image such as a slice and a block) and the information corresponding to the image can be linked to each other during decoding. In other words, the information can be transmitted via a transmission path different from the transmission path used for the encoded data. In addition, the information can be recorded on a recording medium different from the recording medium used for the encoded data (or another recording area of the same recording medium). In addition, the information and the encoded data can be associated with each other in arbitrary units such as multiple frames, one frame or a part of a frame.

The present disclosure can be applied to encoding devices or decoding devices used in the following situations: as in MPEG, H.26x, etc., a bit stream compressed by orthogonal transform such as discrete cosine transform and motion compensation is received via network media such as satellite broadcasting, cable TV, the Internet and mobile phones, or the bit stream is processed on a storage medium such as an optical disc, a magnetic disk and a flash memory.

In addition, in this specification, a system means a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are included in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network are a system, and a single device including multiple modules in a single housing is also a system.

Furthermore, the effects described in this specification are merely exemplary effects and not limitative effects, and any other effects may be produced.

Furthermore, the embodiments of the present disclosure are not limited to the embodiments described above, and various modifications may be made thereto without departing from the gist of the present disclosure.

For example, in the first and second embodiments, the position of the image identification information of the current image registered in the reference image list may not be fixed. In addition, in the first and third embodiments, the image type of the current image set in the reference image list may be STRP.

Furthermore, in the second embodiment, as in the third embodiment, the image identification information of the current image may be registered not in the reference image list but in the temporary list.

Furthermore, in the third embodiment, as in the second embodiment, by changing the method of generating the prediction vector list when the type of the reference image of the current image is Intra BC, the encoding efficiency can be improved when Intra BC and inter-frame encoding are standardized.

Furthermore, for example, the present disclosure may have a configuration of cloud computing in which a plurality of devices share one function and cooperate via a network to perform processing.

Furthermore, the steps described in the above flowcharts may be performed by one device, or may be shared and performed by a plurality of devices.

Furthermore, in the case where one step includes a plurality of processing steps, the plurality of processing steps in one step may be performed by one device, or may be shared and performed by a plurality of devices.

The present disclosure may also have the following configurations.

(1) An image processing device comprising:

a prediction vector generating section that, when encoding a current motion vector of a current block used for prediction using correlation within a picture, sets a candidate block corresponding to a candidate of the reference motion vector to be unavailable if a type of a reference image of the current block and a type of a reference image of the candidate block are different from each other, and generates a prediction vector of the current motion vector by using the reference motion vector, the reference motion vector being referenced when generating the prediction vector of the current motion vector; and

The differential vector generating unit generates a differential vector between the current motion vector and the prediction vector generated by the prediction vector generating unit.

(2) The image processing device according to (1), further comprising:

A setting unit sets the type of the reference image of the current block to a long-term reference image.

(3) The image processing apparatus according to (1), wherein

In a case where the image of the candidate block and the reference image of the candidate block are different from each other, the prediction vector generation part determines that the type of the reference image of the current block and the type of the reference image of the candidate block are different from each other.

(4) The image processing apparatus according to any one of (1) to (3), further comprising:

The list creating unit registers the image of the current block in a predetermined position of the list of candidates for the reference image of the current block.

(5) The image processing apparatus according to (4), wherein:

The predetermined position is the beginning.

(6) The image processing apparatus according to (4) or (5), wherein:

The list creating unit rearranges the order of the reference image candidates in the list before registering the image of the current block.

(7) The image processing apparatus according to (4) or (5), wherein:

The list creating unit rearranges the order of candidates for reference images other than the image of the current block in the list after registering the image of the current block.

(8) An image processing method for an image processing device, the image processing method comprising:

a prediction vector generating step of, when encoding a current motion vector of a current block used for prediction using correlation within a picture, setting a candidate block to be unavailable when a type of a reference image of the current block and a type of a reference image of a candidate block corresponding to a candidate of the reference motion vector are different from each other, and generating a prediction vector of the current motion vector by using the reference motion vector, the reference motion vector being referred to when generating the prediction vector of the current motion vector; and

A differential vector generating step generates a differential vector between the current motion vector and the prediction vector generated by the processing of the prediction vector generating step.

(9) An image processing device comprising:

a prediction vector generating section that, when decoding a current motion vector of a current block used for prediction using correlation within a picture, sets a reference image of the current block and a reference image of a candidate block corresponding to a candidate of the reference motion vector to be unavailable when the type of the reference image of the current block and the type of the reference image of the candidate block are different from each other, and generates a prediction vector of the current motion vector by using the reference motion vector, the reference motion vector being referred to when generating the prediction vector of the current motion vector; and

The motion vector generating section adds a differential vector between the current motion vector and the predicted vector and the predicted vector generated by the predicted vector generating section to generate the current motion vector.

(10) The image processing device according to (9), further comprising:

A setting unit sets the type of the reference image of the current block to a long-term reference image.

(11) The image processing apparatus according to (9), wherein

In a case where the image of the candidate block and the reference image of the candidate block are different from each other, the prediction vector generation part determines that the type of the reference image of the current block and the type of the reference image of the candidate block are different from each other.

(12) The image processing apparatus according to any one of (9) to (11), further comprising:

The list creating unit registers the image of the current block in a predetermined position of the list of candidates for the reference image of the current block.

(13) The image processing apparatus according to (12), wherein

The predetermined position is the beginning.

(14) The image processing apparatus according to (12) or (13), wherein:

The list creating unit rearranges the order of the reference image candidates in the list before registering the image of the current block.

(15) The image processing apparatus according to (12) or (13), wherein:

The list creating unit rearranges the order of candidates for reference images other than the image of the current block in the list after registering the image of the current block.

(16) An image processing method for an image processing device, the image processing method comprising:

a prediction vector generating step of, when decoding a current motion vector of a current block used for prediction using correlation within a picture, setting a reference image of the current block and a reference image of a candidate block corresponding to a candidate of the reference motion vector to be unavailable in a case where the type of the reference image of the current block and the type of the reference image of the candidate block are different from each other, and generating a prediction vector of the current motion vector by using the reference motion vector, the reference motion vector being referred to when generating the prediction vector of the current motion vector; and

A motion vector generating step of adding a differential vector between the current motion vector and the predicted vector and the predicted vector generated by the process of the predicted vector generating step, and generating the current motion vector.

Reference Signs List

50 encoding equipment

88 List Creation Department

90 Prediction vector generation unit

91 Differential vector generation unit

110 decoding device

144 Prediction vector generation unit

145 Motion Vector Generation Unit

146 List Creation Department

Claims (6)

1. An image processing apparatus, comprising: The prediction vector generation unit is configured to: When encoding the current motion vector of the current block used for prediction based on in-frame correlation, if the type of the reference image of the current block is different from the type of the reference image of the candidate block corresponding to the reference motion vector, the candidate block is set to unavailable. When both the reference image type of the current block and the reference image type of the candidate block are Short-Term Reference Images (STRP), a prediction vector for the current motion vector is generated by scaling the reference motion vector. When both the reference image type of the current block and the reference image type of the candidate block are Intra Block Copy (Intra BC), a prediction vector for the current motion vector is generated by setting the reference motion vector as the prediction vector without scaling. When both the reference image type of the current block and the reference image type of the candidate block are Long-Term Reference Images (LTRP), the prediction vector of the current motion vector is generated by setting the reference motion vector as the prediction vector without scaling, and the reference motion vector is referenced when generating the prediction vector of the current motion vector. The list creation unit is configured to register the image of the current block in a predetermined position in a candidate list of reference images of the current block, wherein the predetermined position is the beginning; A difference vector generation unit is configured to use the list to generate a difference vector between the current motion vector and the prediction vector generated by the prediction vector generation unit; and An encoding unit is configured to encode the difference vector. 2. The image processing apparatus according to claim 1, further comprising: The setting unit sets the type of the reference image for the current block to a long-term reference image. 3. The image processing apparatus according to claim 1, wherein, When the image of a candidate block is different from the reference image of the candidate block, the prediction vector generation unit determines that the type of the reference image of the current block is different from the type of the reference image of the candidate block. 4. The image processing apparatus according to claim 1, wherein, Before registering the image for the current block, the list creation department rearranges the order of the candidate reference images in the list. 5. The image processing apparatus according to claim 1, wherein, After registering the image for the current block, the list creation department rearranges the order of candidate reference images (excluding the image for the current block) in the list. 6. An image processing method for an image processing apparatus, the image processing method comprising: In the prediction vector generation step, when encoding the current motion vector of the current block used for prediction based on in-frame correlation, if the type of the reference image of the current block is different from the type of the reference image of the candidate block corresponding to the reference motion vector, the prediction vector generation step sets the candidate block to unavailable. If the type of the reference image of the current block and the type of the reference image of the candidate block are both Short-Term Reference Images (STRP), the prediction vector of the current motion vector is generated by scaling the reference motion vector. If the type of the reference image of the current block and the type of the reference image of the candidate block are both Intra Block Copy (Intra BC), the prediction vector of the current motion vector is generated by setting the reference motion vector as the prediction vector without scaling. If the type of the reference image of the current block and the type of the reference image of the candidate block are both Long-Term Reference Images (LTRP), the prediction vector of the current motion vector is generated by setting the reference motion vector as the prediction vector without scaling. The reference motion vector is referenced when generating the prediction vector of the current motion vector. The difference vector generation step generates a difference vector between the current motion vector and the predicted vector generated by the prediction vector generation step; and The list creation step registers the image of the current block in a predetermined position in the candidate list of reference images of the current block, wherein the predetermined position is the beginning.