WO2014097937A1

WO2014097937A1 - Image processing device and image processing method

Info

Publication number: WO2014097937A1
Application number: PCT/JP2013/083154
Authority: WO
Inventors: 碩陸; 佐藤　数史; 央二中神
Original assignee: ソニー株式会社
Priority date: 2012-12-20
Filing date: 2013-12-11
Publication date: 2014-06-26

Abstract

The present technology relates to an image processing device and an image processing method whereby it is possible to improve coding efficiency when a motion vector of an image of a base layer is employed as a candidate of a prediction vector of a motion vector of an image of an enhancement layer. A processing unit uses a motion vector to carry out a correction process of an enhancement image. On the basis of characteristics of an image formed from the base image and the enhancement image, a list registration unit registers, in a list, prediction vector candidates of a motion vector of the enhancement image, which include the motion vector of the base image. The present technology may be applied, as an example, to a coding device whereby a scalable coding is carried out.

Description

Image processing apparatus and image processing method

The present technology relates to an image processing device and an image processing method, and in particular, it is possible to improve encoding efficiency when a motion vector of a base layer image is used as a motion vector prediction vector candidate of an enhancement layer image. The present invention relates to an image processing apparatus and an image processing method.

In recent years, image information is handled as digital data, and MPEG (compressed by orthogonal transform such as discrete cosine transform and motion compensation is used for the purpose of efficient transmission and storage of information. A device compliant with a method such as Moving (Pictures Experts Group) phase) is becoming popular in both information distribution at broadcast stations and information reception in general households.

In particular, the MPEG2 (ISO / IEC 13818-2) system is defined as a general-purpose image encoding system, and is a standard that covers both interlaced and progressively scanned images, standard resolution images, and high-definition images. Widely used in a wide range of applications for consumer and consumer applications. By using the MPEG2 method, for example, a standard resolution interlaced scanning image having 720 × 480 pixels is 4 to 8 Mbps, and a high resolution interlaced scanning image having 1920 × 1088 pixels is 18 to 22 MBps. By assigning a (rate), it is possible to realize a high compression rate and good image quality.

MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but it did not support encoding methods with a lower code amount (bit rate) than MPEG1, that is, a higher compression rate. With the widespread use of mobile terminals, the need for such an encoding system is expected to increase in the future, and the MPEG4 encoding system has been standardized accordingly. Regarding the MPEG4 image coding system, the standard was approved as an international standard in December 1998 as ISO / IEC 449 14496-2.

Furthermore, in recent years, for the purpose of image coding for the initial video conference, The standardization of 26L (ITU-T Q6 / 16 という VCEG) is in progress. H. 26L is known to achieve higher encoding efficiency than the conventional encoding schemes such as MPEG2 and MPEG4, although a large amount of calculation is required for encoding and decoding.

Also, as part of MPEG4 activities, this H.264 Based on 26L, H. Standardization to achieve higher coding efficiency by incorporating functions that are not supported by 26L is performed as JointJModel of Enhanced-Compression Video Coding. This standardization was implemented in March 2003 by H.C. It was internationally standardized under the names of H.264 and MPEG-4® Part 10 (AVC (Advanced Video Coding)).

Furthermore, as an extension, standardization of FRExt® (Fidelity Range Extension) including coding tools necessary for business such as RGB, YUV422, and YUV444, and 8 × 8DCT and quantization matrix specified by MPEG-2. Was completed in February 2005. As a result, the AVC system has become an encoding system that can well express film noise included in movies, and has been used for a wide range of applications such as BD (Blu-ray (registered trademark) Disc).

However, these days, we want to compress images with a resolution of about 4000 x 2000 pixels, which is four times that of high-definition images, or to deliver high-definition images in environments with limited transmission capacity such as the Internet. Needs are growing. For this reason, in the VCEG (Video Coding Expert Group) under the ITU-T, studies on improving the coding efficiency are continuing.

In addition, with the aim of further improving coding efficiency compared to AVC, ITUHEVC (High Efficiency Video Coding) has been established by JCTVC (Joint Collaboration Team Coding), a joint standardization organization of ITU-T and ISO / IEC. The standardization of the encoding method called is being advanced.

In the HEVC method, two motion vector encoding methods, AMVP (Advanced Motion Vector Prediction) and Merge, are defined.

In either encoding method, motion vectors of neighboring PUs (Prediction Units) are regarded as motion vector prediction vector candidates for the processing target PU and registered in the list. A prediction vector is determined from prediction vector candidates, and prediction vector identification information for identifying a prediction vector in the list is transmitted from the encoding device to the decoding device. Note that the prediction vector identification information is information indicating the order of registration in a list given to prediction vector candidates.

Also, in AMVP, the difference between the prediction vector and the motion vector of the PU to be processed is transmitted from the encoding device to the decoding device. On the other hand, in Merge, the prediction vector is the motion vector of the PU to be processed.

As a peripheral PU, a temporally neighboring PU, that is, a region having the same position in the screen as the processing target PU and having a different time, or a PU around that region (hereinafter referred to as a temporal peripheral PU), spatially There are peripheral PUs, that is, peripheral PUs in the same screen as the processing target PU (hereinafter referred to as spatial peripheral PUs).

Specifically, for example, as shown in FIG. 1, the time-peripheral PU includes a block C0 that is a PU having the same position of the center of the PU1 to be processed and the center of the screen at different times, and PU1 and the screen in the screen. This is block H, which is the lower right PU in the region having the same position and different time. Further, as shown in FIG. 1, the space peripheral PU is a block A1 and a block A2 that are two PUs on the lower left side of PU1, and blocks B0 to B2 that are three upper PUs.

FIG. 2 is a diagram for explaining a method of registering a prediction vector candidate in a list.

Here, a PU having the same processing target PU, reference image specifying information (ref_idx) for specifying a reference image, and a prediction direction are called VEC1 PUs, and the reference image specifying information is the same, but the prediction directions are different. PU is called VEC2 PU. Further, a PU to be processed and a PU having different reference image specifying information but having the same prediction direction are referred to as a VEC3 PU, and a PU having both reference image specifying information and a prediction direction being referred to as a VEC4 PU.

In AMVP, motion vectors of up to two spatial surrounding PUs are registered in the list as prediction vector candidates. Specifically, as shown in FIG. 2A, in AMVP, it is first determined whether or not there is a motion vector of block A0 and block A1 of VEC1, and then the determination is VEC2, VEC3, VEC4. Are performed in order. This determination is terminated when it is determined that it exists, and the motion vector of the existing block A0 or block A1 is registered in the list as a prediction vector candidate.

Then, it is determined in order whether or not motion vectors of the blocks B0 to B2 of VEC1 exist, and then the determination is performed in order for VEC2, VEC3, and VEC4. This determination is terminated when it is determined that it exists, and any one motion vector of the existing blocks B0 to B2 is registered in the list as a prediction vector candidate.

Next, when there is a motion vector of block H that is a time peripheral PU, the motion vector of block H is registered in the list as a candidate for a prediction vector, and when there is no motion vector of block H, the motion vector of block C0 is The prediction vector candidate is registered in the list. Finally, when the number of prediction vector candidates is less than a predetermined number, 0 as a motion vector is registered in the list as a prediction vector candidate.

In Merge, motion vectors of up to five spatial surrounding PUs are registered in the list as prediction vector candidates. Specifically, as shown in FIG. 2B, first, motion vectors existing among the motion vectors of the block A0 and the block A1, and the block B0 and the block B1 are set as prediction vector candidates. At this time, if at least one of the motion vectors of the block A0 and the block A1 and the block B0 and the block B1 does not exist, the motion vector of the block B2 is registered in the list as a prediction vector candidate instead of the non-existing motion vector. The

Next, when there is a motion vector of block H that is a time peripheral PU, the motion vector of block H is registered in the list as a candidate for a prediction vector, and when there is no motion vector of block H, the motion vector of block C0 is The prediction vector candidate is registered in the list.

Next, when the number of prediction vector candidates is less than a predetermined number and the slice type is a B slice, a combination of prediction vector candidates for forward prediction and backward prediction is registered in the list. Finally, when the number of prediction vector candidates is less than a predetermined number, 0 as a motion vector is registered in the list as a prediction vector candidate.

By the way, image encoding methods such as MPEG-2 and AVC have a scalability function for encoding an image in a hierarchical manner. According to the scalability function, it is possible to transmit encoded data according to the processing capability on the decoding side without performing a transcoding process.

Specifically, for example, only a coded stream of an image of a base layer (base layer) that is a base layer can be transmitted to a terminal with low processing capability such as a mobile phone. On the other hand, an encoded stream of an image of a base layer and an enhancement layer (enhancement layer) that is a layer other than the base layer may be transmitted to a terminal having high processing capability such as a television receiver or a personal computer. it can.

Such a scalability function is also provided in the HEVC system. When encoding is performed using the scalability function, the above-described motion vector encoding is performed in each layer. Therefore, it is considered to improve the encoding efficiency by using the motion vector of the base layer image as a prediction vector candidate when encoding the enhancement layer motion vector (see, for example, Non-Patent Document 1).

However, the order of registration in the motion vector list of base layer images is fixed. Accordingly, although the motion vector of the base layer image is likely to be a prediction vector, the prediction vector identification information of the motion vector is large or low, but the prediction vector of the motion vector is low. The identification information is small. As a result, encoding efficiency is poor.

The present technology has been made in view of such a situation, and can improve coding efficiency when a motion vector of a base layer image is used as a motion vector prediction vector candidate of an enhancement layer image. It is something that can be done.

An image processing apparatus according to an aspect of the present technology, based on a feature of a processing unit that performs compensation processing on an image of a first layer of an image having a hierarchical structure using a motion vector, and an image having the hierarchical structure, An image processing apparatus comprising: a list registration unit that registers a motion vector prediction vector candidate of a first layer image including a motion vector of a second layer image in a list.

The image processing method according to one aspect of the present technology corresponds to the image processing apparatus according to one aspect of the present technology.

In one aspect of the present technology, compensation processing is performed on an image in a first layer of an image having a hierarchical structure using a motion vector, and the second layer is based on the characteristics of the image having the hierarchical structure. The motion vector prediction vector candidates including the motion vector of the first layer are registered in the list.

Note that the image processing apparatus according to one aspect of the present technology can be realized by causing a computer to execute a program.

Also, in order to realize the image processing apparatus according to one aspect of the present technology, a program to be executed by a computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

According to one aspect of the present technology, encoding efficiency can be improved when a motion vector of a base layer image is used as a candidate for a motion vector prediction vector of an enhancement layer image.

It is a figure which shows an example of the conventional periphery PU. It is a figure explaining the registration method to the list of prediction vector candidates. It is a figure explaining a scalability function. It is a block diagram which shows the structural example of one Embodiment of the encoding apparatus to which this technique is applied. It is a block diagram which shows the structural example of the enhancement encoding part of FIG. It is a figure which shows the structural example of the syntax of SPS. FIG. 6 is a block diagram illustrating a configuration example of an encoding unit in FIG. 5. FIG. 8 is a block diagram illustrating a configuration example of a motion prediction / compensation unit in FIG. 7. It is a figure which shows the example of PU corresponding to the prediction vector candidate. It is a figure which shows the example of the size of PU. It is a figure explaining the determination method of the registration order of the prediction vector candidate. 6 is a flowchart for explaining generation processing of the encoding device in FIG. 4. 13 is a flowchart for explaining details of the encoding process of FIG. 12. It is a flowchart explaining the detail of the list production | generation process of FIG. It is a flowchart explaining the detail of the list production | generation process of FIG. It is a block diagram which shows the structural example of the enhancement decoding part of FIG. It is a block diagram which shows the structural example of the decoding part of FIG. It is a block diagram which shows the structural example of the motion compensation part of FIG. It is a flowchart explaining the image generation process of the decoding apparatus of FIG. It is a flowchart explaining the detail of the decoding process of FIG. It is a figure explaining the other determination method of the registration order of the candidate of a prediction vector. It is a flowchart explaining the list production | generation process in case registration order is determined by the determination method of FIG. It is a figure explaining the registration method of the list based on the size of CU. It is a figure explaining the registration method of the list based on the kind of scalability function. It is a figure explaining the registration method of the list based on a quantization parameter. It is a figure explaining selection of hierarchy periphery PU and time periphery PU. It is a figure which shows the structural example of the syntax of PPS. It is a figure which shows the structural example of the syntax of a slice header. It is a figure which shows the example of a multiview image encoding system. The other example of the encoding by a Scalability function is shown. It is a block diagram which shows the structural example of the hardware of a computer. It is a figure which shows the schematic structural example of the television apparatus to which this technique is applied. It is a figure which shows the schematic structural example of the mobile telephone to which this technique is applied. It is a figure which shows the schematic structural example of the recording / reproducing apparatus to which this technique is applied. It is a figure which shows the schematic structural example of the imaging device to which this technique is applied. It is a block diagram which shows an example of scalable encoding utilization. It is a block diagram which shows the other example of scalable encoding utilization. It is a block diagram which shows the further another example of scalable encoding utilization.

In the following, it is assumed that there is one enhancement layer unless otherwise specified, but the number of enhancement layers may be plural.

<Explanation of scalability function>
FIG. 3 is a diagram for explaining the scalability function.

As shown in FIG. 3, SNRabilityscalability is a scalability function that encodes an image layered with SNR (signal-noise ratio). Specifically, in SNR scalability, a low SNR image is encoded as a base layer image, and a difference image between a high SNR image and a low SNR image is encoded as an enhancement layer image.

Accordingly, the encoding device transmits only the encoded data of the base layer image to the decoding device with low processing capability, so that the decoding device can generate a low SNR image, that is, a low quality image. it can. Further, the encoding device transmits the encoded data of the base layer and enhancement layer images to the decoding device having high processing capability, so that the decoding device decodes and synthesizes the base layer and enhancement layer images. A high SNR image, that is, a high-quality image can be generated.

As shown in FIG. 3B, spatial scalability is a scalability function that encodes an image by layering it at a spatial resolution. Specifically, in spatial scalability, a low resolution image is encoded as a base layer image, and a difference image between the high resolution image and the low resolution image is encoded as an enhancement layer image.

Therefore, the encoding device transmits only the encoded data of the base layer image to the decoding device with low processing capability, so that the decoding device can generate a low-resolution image. Further, the encoding device transmits the encoded data of the base layer and enhancement layer images to the decoding device having high processing capability, so that the decoding device decodes and synthesizes the base layer and enhancement layer images. High-resolution images can be generated.

Although illustration is omitted, there are other scalability functions besides SNRSscalability and spatial scalability.

For example, as a scalability function, there is also temporal-scalability for hierarchizing and encoding an image by a frame rate, and bit-depth-scalability for encoding an image by hierarchizing by a bit number. In addition, as a scalability function, there is also a chroma-scalability for hierarchizing and encoding an image in a color difference signal format.

<One embodiment>
(Configuration example of one embodiment of encoding device)
FIG. 4 is a block diagram illustrating a configuration example of an embodiment of an encoding device to which the present technology is applied.

4 includes a base encoding unit 11, an enhancement encoding unit 12, a synthesizing unit 13, and a transmission unit 14. The encoding apparatus 10 encodes an image using a scalability function in accordance with a scheme conforming to the HEVC scheme.

A base layer image (hereinafter referred to as a base image) is input to the base encoding unit 11 of the encoding device 10 from the outside. The base encoding unit 11 is configured in the same manner as a conventional HEVC encoding device, and encodes a base image using the HEVC method. However, the base encoding unit 11 supplies a motion vector used at the time of inter encoding of the base image to the enhancement encoding unit 12. The base encoding unit 11 supplies an encoded stream including encoded data, SPS, PPS, and the like obtained as a result of encoding to the synthesizing unit 13 as a base stream.

The enhancement coding unit 12 receives an enhancement layer image (hereinafter referred to as an enhancement image) from the outside. The enhancement encoding unit 12 encodes the enhancement image by a method according to the HEVC method. Further, the enhancement encoding unit 12 uses the motion vector of the base image supplied from the base encoding unit 11 to encode the motion vector used at the time of inter-encoding of the enhancement image by the AMVP method, and obtains motion vector information. Generate.

The enhancement encoding unit 12 generates an encoded stream by adding motion vector information or the like to the encoded data of the enhancement image, and supplies the encoded stream to the synthesizing unit 13 as an enhancement stream.

The synthesizing unit 13 synthesizes the base stream supplied from the base encoding unit 11 and the enhancement stream supplied from the enhancement encoding unit 12 to generate an encoded stream of all layers. The synthesizing unit 13 supplies the encoded stream of all layers to the transmission unit 14.

The transmission unit 14 transmits the encoded stream of all layers supplied from the synthesis unit 13 to a decoding device described later.

In addition, although the encoding apparatus 10 shall transmit the encoding stream of all the layers here, it can also transmit only a base stream as needed.

(Configuration example of enhancement encoding unit)
FIG. 5 is a block diagram illustrating a configuration example of the enhancement encoding unit 12 of FIG.

The enhancement encoding unit 12 in FIG. 5 includes a setting unit 21 and an encoding unit 22.

The setting unit 21 of the enhancement encoding unit 12 sets a parameter set such as SPS and PPS and supplies the parameter set to the encoding unit 22.

The encoding unit 22 encodes an enhancement image input from the outside by a method according to the HEVC method, and generates encoded data. At this time, the encoding unit 22 uses the motion vector from the base encoding unit 11 to encode a motion vector used for inter prediction of the enhancement image by the AMVP method, and generates motion vector information.

The encoding unit 22 adds motion vector information or the like as header information to the encoded data of the enhancement image. The encoding unit 22 adds the parameter set supplied from the setting unit 21 to the encoded data of the enhancement image to which the header information is added, and generates an enhancement stream. The encoding unit 22 supplies the enhancement stream to the synthesis unit 13 in FIG.

(Configuration example of SPS syntax of enhancement stream)
FIG. 6 is a diagram illustrating a configuration example of SPS syntax set by the setting unit 21 of FIG.

As shown in the 23rd line of FIG. 6, the SPS includes a base vector flag (sps_enable_BLMV_flag) (identification information) for identifying that the motion vector of the base image is a prediction vector candidate. The base vector flag is, for example, 1 when representing that the motion vector of the base image is a prediction vector candidate, and is 0 when representing that the motion vector of the base image is not a prediction vector candidate.

(Configuration example of encoding unit)
FIG. 7 is a block diagram illustrating a configuration example of the encoding unit 22 of FIG.

7 includes an A / D conversion unit 101, a screen rearrangement buffer 102, a calculation unit 103, an orthogonal transformation unit 104, a quantization unit 105, a lossless encoding unit 106, and a storage buffer 107. The encoding unit 22 includes an inverse quantization unit 108, an inverse orthogonal transform unit 109, a calculation unit 110, a filter 111, a decoded picture buffer 112, a selection unit 113, an intra prediction unit 114, a motion prediction / compensation unit 115, and a prediction. An image selection unit 116 is included.

Specifically, the A / D conversion unit 101 of the encoding unit 22 performs A / D conversion on the input enhancement image, and supplies the image that is the converted digital data to the screen rearrangement buffer 102 for storage. . The screen rearrangement buffer 102 rearranges the stored frames in the display order in the order of frames for encoding in accordance with GOP (Group Of Picture). The screen rearrangement buffer 102 supplies the image with the rearranged frame order to the arithmetic unit 103.

Also, the screen rearrangement buffer 102 supplies the image in which the frame order is rearranged to the intra prediction unit 114 and the motion prediction / compensation unit 115.

The calculation unit 103 subtracts the predicted image supplied from the intra prediction unit 114 or the motion prediction / compensation unit 115 via the predicted image selection unit 116 from the image read from the screen rearrangement buffer 102. The calculation unit 103 outputs the difference information obtained as a result to the orthogonal transform unit 104.

For example, when intra coding is performed, the calculation unit 103 subtracts the prediction image supplied from the intra prediction unit 114 from the image read from the screen rearrangement buffer 102. Further, when inter coding is performed, the arithmetic unit 103 subtracts the predicted image supplied from the motion prediction / compensation unit 115 from the image read from the screen rearrangement buffer 102.

The orthogonal transform unit 104 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 103. Note that this orthogonal transformation method is arbitrary. The orthogonal transform unit 104 supplies the transform coefficient to the quantization unit 105.

The quantization unit 105 quantizes the transform coefficient supplied from the orthogonal transform unit 104. The quantization unit 105 sets a quantization parameter based on information on the code amount target value, and performs the quantization. Note that this quantization method is arbitrary. The quantization unit 105 supplies the quantized transform coefficient to the lossless encoding unit 106.

The lossless encoding unit 106 losslessly encodes the transform coefficient quantized by the quantization unit 105 using an arbitrary encoding method, and generates encoded data. Further, the lossless encoding unit 106 acquires intra prediction information including information indicating an intra prediction mode from the intra prediction unit 114, and moves inter prediction information including information indicating an inter prediction mode, motion vector information, and the like. Obtained from the prediction / compensation unit 115. Further, the lossless encoding unit 106 acquires filter coefficients used in the filter 111 and the like.

The lossless encoding unit 106 performs lossless encoding of these various types of information using an arbitrary encoding method, and uses (multiplexes) a part of the header information of the encoded data. The lossless encoding unit 106 supplies the encoded data in which the header information is multiplexed to the accumulation buffer 107 for accumulation.

Examples of the encoding method of the lossless encoding unit 106 include variable length encoding or arithmetic encoding. Examples of variable length coding include H.264. CAVLC (Context-Adaptive Variable Length Coding) defined in the H.264 / AVC format. Examples of arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

The accumulation buffer 107 temporarily holds the encoded data supplied from the lossless encoding unit 106. The accumulation buffer 107 reads out the encoded data held at a predetermined timing, and supplies it as an enhancement stream to the synthesis unit 13 in FIG. 4 together with the parameter set supplied from the setting unit 21 in FIG.

Also, the transform coefficient quantized by the quantization unit 105 is also supplied to the inverse quantization unit 108. The inverse quantization unit 108 inversely quantizes the quantized transform coefficient by a method corresponding to the quantization by the quantization unit 105. The inverse quantization method may be any method as long as it is a method corresponding to the quantization processing by the quantization unit 105. The inverse quantization unit 108 supplies the obtained transform coefficient to the inverse orthogonal transform unit 109.

The inverse orthogonal transform unit 109 performs inverse orthogonal transform on the transform coefficient supplied from the inverse quantization unit 108 by a method corresponding to the orthogonal transform process by the orthogonal transform unit 104. The inverse orthogonal transform method may be any method as long as it corresponds to the orthogonal transform processing by the orthogonal transform unit 104. The inversely orthogonally transformed output (difference information restored locally) is supplied to the calculation unit 110.

The calculation unit 110 converts the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 109, that is, locally restored difference information, into the intra prediction unit 114 or the motion prediction / compensation unit 115 via the predicted image selection unit 116. Are added to the predicted image to obtain a locally decoded image (hereinafter referred to as a reconstructed image). The reconstructed image is supplied to the filter 111 or the decoded picture buffer 112.

The filter 111 includes a deblock filter, an adaptive loop filter, and the like, and appropriately performs a filtering process on the reconstructed image supplied from the calculation unit 110. For example, the filter 111 removes block distortion of the reconstructed image by performing deblocking filter processing on the reconstructed image. Further, for example, the filter 111 improves the image quality by performing loop filter processing using a Wiener filter on the deblock filter processing result (reconstructed image from which block distortion has been removed). Do.

Note that the filter 111 may perform arbitrary filter processing on the reconstructed image. In addition, the filter 111 can supply information such as filter coefficients used for the filter processing to the lossless encoding unit 106 and perform lossless encoding as necessary.

The filter 111 supplies a filter processing result (hereinafter referred to as a decoded image) to the decoded picture buffer 112.

The decoded picture buffer 112 stores the reconstructed image supplied from the calculation unit 110 and the decoded image supplied from the filter 111.

The decoded picture buffer 112 supplies the stored reconstructed image to the intra prediction unit 114 via the selection unit 113 at a predetermined timing or based on a request from the outside such as the intra prediction unit 114. In addition, the decoded picture buffer 112 receives a stored decoded image at a predetermined timing or based on an external request from the motion prediction / compensation unit 115 or the like via the selection unit 113. 115.

The selection unit 113 indicates the supply destination of the reconstructed image or decoded image output from the decoded picture buffer 112. Specifically, when intra coding is performed, the selection unit 113 reads a reconstructed image that has not been filtered from the decoded picture buffer 112, and an image of a peripheral region (peripheral image) located around the prediction target region. Is supplied to the intra prediction unit 114.

Also, when inter coding is performed, the selection unit 113 reads the decoded image that has been filtered from the decoded picture buffer 112, and supplies the decoded image to the motion prediction / compensation unit 115 as a reference image.

The intra prediction unit 114, when acquiring a peripheral image from the decoded picture buffer 112, performs intra prediction using the pixel value of the peripheral image to generate a prediction image using the PU as a processing unit. The intra prediction unit 114 performs this intra prediction in all candidate intra prediction modes.

The intra prediction unit 114 evaluates the cost function value of each prediction image using the prediction image generated by the intra prediction of all candidate intra prediction modes and the input image supplied from the screen rearrangement buffer 102. Select the optimal intra prediction mode. Then, the intra prediction unit 114 supplies the predicted image and the cost function value generated in the optimal intra prediction mode to the predicted image selection unit 116.

In addition, when the selection is notified from the prediction image selection unit 116, the intra prediction unit 114 appropriately supplies intra prediction information including information related to intra prediction such as an optimal intra prediction mode to the lossless encoding unit 106, and Make it.

The motion prediction / compensation unit 115 reads out the decoded image stored in the decoded picture buffer 112 as a reference image. The motion prediction / compensation unit 115 uses the input image supplied from the screen rearrangement buffer 102 and the reference image as a unit of processing, and performs motion prediction using inter-time correlation as inter prediction.

The motion prediction / compensation unit 115 performs a compensation process according to the motion vector detected as a result of the inter prediction, and generates a predicted image. The motion prediction / compensation unit 115 performs such inter prediction and compensation processing in all candidate inter prediction modes.

The motion prediction / compensation unit 115 evaluates the cost function value of each prediction image using the prediction images in all candidate inter prediction modes and the input image supplied from the screen rearrangement buffer 102, and determines the optimum value. Select inter prediction mode. Then, the motion prediction / compensation unit 115 supplies the predicted image and the cost function value generated in the optimal inter prediction mode to the predicted image selection unit 116.

In addition, when the selection is notified from the prediction image selection unit 116, the motion prediction / compensation unit 115, based on the motion vector from the base encoding unit 11 and the base vector flag included in the SPS from the setting unit 21, A motion vector corresponding to the predicted image generated in the optimal inter prediction mode is encoded by the AMVP method. The motion prediction / compensation unit 115 supplies the motion vector information obtained as a result and inter prediction information including information related to inter prediction such as the optimal inter prediction mode to the lossless encoding unit 106 and performs lossless encoding.

The predicted image selection unit 116 selects the supply source of the predicted image supplied to the calculation unit 103 and the calculation unit 110 based on the cost function values supplied from the intra prediction unit 114 and the motion prediction / compensation unit 115.

Specifically, when the cost function value supplied from the intra prediction unit 114 is smaller than the cost function value supplied from the motion prediction / compensation unit 115, the predicted image selection unit 116 uses the intra prediction unit as a source of the predicted image. 114 is selected. Thereby, the prediction image supplied from the intra estimation part 114 is supplied to the calculating part 103 and the calculating part 110, and intra encoding is performed.

On the other hand, when the cost function value supplied from the motion prediction / compensation unit 115 is smaller than the cost function value supplied from the intra prediction unit 114, the prediction image selection unit 116 serves as a prediction image supply source. Select. Thereby, the predicted image supplied from the motion prediction / compensation unit 115 is supplied to the calculation unit 103 and the calculation unit 110, and inter coding is performed. The predicted image selection unit 116 notifies the selected supply source of the selection.

(Configuration example of motion prediction / compensation unit)
FIG. 8 is a block diagram illustrating a configuration example of the motion prediction / compensation unit 115 of FIG.

8 includes a processing unit 131, a prediction information buffer 132, a list registration unit 133, a vector buffer 134, and a vector encoding unit 135.

The processing unit 131 of the motion prediction / compensation unit 115 reads out the decoded image stored in the decoded picture buffer 112 of FIG. 7 as a reference image. The processing unit 131 uses the input image and the reference image supplied from the screen rearrangement buffer 102 to perform inter prediction using the PU as a processing unit. The processing unit 131 performs compensation processing according to the motion vector detected as a result of inter prediction, and generates a predicted image. The processing unit 131 performs such inter prediction and compensation processing in all candidate inter prediction modes.

The processing unit 131 evaluates the cost function value of each prediction image using the prediction images and input images of all candidate inter prediction modes, and selects an optimal inter prediction mode. Then, the processing unit 131 supplies the predicted image and the cost function value generated in the optimal inter prediction mode to the predicted image selection unit 116.

Further, when the selection is notified from the prediction image selection unit 116, the processing unit 131 displays the PU size, prediction direction, motion vector, and reference image specifying information corresponding to the prediction image generated in the optimal inter prediction mode. The prediction information is supplied to the prediction information buffer 132 as prediction information. In this case, the processing unit 131 supplies a motion vector corresponding to the predicted image generated in the optimal inter prediction mode to the vector encoding unit 135. Further, in this case, the processing unit 131 supplies the optimal inter prediction mode, the reference image specifying information, and the prediction direction representing the PU size to the lossless encoding unit 106 in FIG. 7 as inter prediction information.

The prediction information buffer 132 stores the prediction information supplied from the motion prediction / compensation unit 115.

The list registration unit 133 determines whether or not to use the motion vector of the base image as a prediction vector candidate based on the base vector flag included in the SPS supplied from the setting unit 21 of FIG.

The list registration unit 133 reads the size of the PU from the prediction information buffer 132 when the motion vector of the base image is a prediction vector candidate. Then, based on the PU size, the list registration unit 133 hierarchically predicts motion vectors of peripheral peripheral PUs (hereinafter referred to as hierarchical peripheral PUs), spatial peripheral PU motion vectors, and temporal peripheral PU motion vectors. The order of registration as candidates is determined.

On the other hand, when the list registration unit 133 does not use the motion vector of the base image as a prediction vector candidate, the list registration unit 133 determines the registration order as the motion vector candidate of the spatial peripheral PU and the prediction vector of the motion vector of the temporal peripheral PU.

The list registration unit 133 registers motion vector identification information, which is identification information of motion vectors that are prediction vector candidates, in the list in the determined registration order.

Note that the registration method of the motion vector of the spatial peripheral PU and the motion vector of the temporal peripheral PU is the same as the method described in FIG. 2, and the motion vector of the spatial peripheral PU is stored in the prediction information buffer 132. The prediction direction of the surrounding spatial PU and the PU to be processed and the reference image specifying information are used.

Also, when the number of motion vector identification information registered in the list is less than a predetermined number, the list registration unit 133 registers motion vector identification information of a motion vector that is 0 in the list. Then, the list registration unit 133 supplies the generated list to the vector encoding unit 135. Note that the motion vector identification information in the list is given prediction vector identification information in the order of registration.

The vector buffer 134 stores the motion vector supplied from the base encoding unit 11.

The vector encoding unit 135 reads the motion vector identified by the motion vector identification information registered in the list supplied from the list registration unit 133 from the prediction information buffer 132 or the vector buffer 134 as a prediction vector candidate. The vector encoding unit 135 determines the prediction vector by comparing the motion vector supplied from the processing unit 131 with the prediction vector candidate. Specifically, the vector encoding unit 135 selects a prediction vector candidate having a small difference from the motion vector supplied from the processing unit 131 as a prediction vector.

The vector encoding unit 135 encodes the motion vector by generating a difference between the motion vector supplied from the processing unit 131 and the prediction vector as motion vector information. The vector encoding unit 135 supplies motion vector information and prediction vector identification information to the lossless encoding unit 106 as inter prediction information.

(Example of PU corresponding to prediction vector candidate)
FIG. 9 is a diagram illustrating an example of PUs corresponding to prediction vector candidates.

In the example of FIG. 9, the spatial peripheral PU corresponding to the prediction vector candidate is one of the block A1 or the block A2 and the blocks B0 to B2, as in the case of FIG. Further, the time periphery PU is the block C0 or the block H as in the case of FIG. Furthermore, the hierarchical peripheral PU is an area where the position in the screen is the same as the PU to be processed or the PU of the base image around the area. In the example of FIG. 9, the positions of the center of the PU 151 and the center in the screen Are the blocks ColBL that are PUs of the base images having the same.

(Example of PU size)
FIG. 10 is a diagram illustrating an example of a PU size.

PU is set by dividing CU. Therefore, for example, as shown in FIG. 10, the CU 170 can be used as the PU 171 as it is. In this case, the size of the PU 171 is the same as the size of the CU 170.

Also, the CU 170 can be divided into upper and lower halves to form PU 172-1 and PU 172-2, or divided into left and right halves to form PU 173-1 and PU 173-2. In this case, the size of PU172-1, PU172-2, PU173-1, and PU173-2 is half of the size of CU170. Further, the CU 170 can be divided into four parts in the vertical and horizontal directions to form PUs 174-1 to 174-4. In this case, the sizes of the PUs 174-1 to 174-4 are 1/4 times the size of the CU 170.

Also, 2/3 on the left side of CU170 can be PU175-1, and 1/3 on the right side can be PU175-2. 1/3 on the left side of CU170 can be PU176-1, and 2/3 on the right side can be PU175-1. It can also be PU176-2. Further, the upper 2/3 of the CU 170 can be the PU 177-1 and the lower 1/3 can be the PU 177-2, or the upper 1/3 of the CU 170 can be the PU 178-1, and the lower 2 / 3 can also be PU178-2. In these cases, the size of PU175-1, PU176-2, PU177-1, and PU178-2 is 2/3 times that of CU170, and PU175-2, PU176-1, PU177-2 and PU178-1 The size is 1/3 times that of CU170.

The processing unit 131 performs inter prediction and compensation processing in the inter prediction mode representing the size of the PU divided as described above, and generates a prediction image. Then, the processing unit 131 evaluates the cost function value of each predicted image and selects an optimal inter prediction mode.

At this time, if there is motion in the entire CU, that is, if the image of the entire CU is uniform, there is a high possibility that the inter prediction mode representing a large size PU is selected as the optimal inter prediction mode. On the other hand, when there is a partial movement in the CU, that is, when the CU image is a fine image, there is a high possibility that an inter prediction mode representing a small size PU is selected as the optimal inter prediction mode.

(Method for determining the registration order of prediction vector candidates)
FIG. 11 is a diagram illustrating a method for determining the registration order of prediction vector candidates by the list registration unit 133 in FIG. 8.

As illustrated in FIG. 11, when the list registration unit 133 determines that the motion vector of the base image is a prediction vector candidate, the list registration unit 133 sets the motion vector registration order of the block ColBL based on the size of the PU to be processed. Or at the end.

Specifically, when the size of the PU to be processed is equal to or larger than a predetermined size, the image of the entire CU is uniform, and the correlation between the motion vectors of the base image and the enhancement image is relatively strong. Therefore, in this case, the registration order of the motion vector of the block ColBL is determined at the head. That is, the prediction vector identification information of the motion vector of the block ColBL is the minimum value.

The registration order after the motion vector of the block ColBL is the same as that in the conventional case shown in FIG. That is, the registration order is determined so that the motion vector of the block A0 or the block A1, the motion vector of any of the blocks B0 to B2, and the motion vector of the block H or the block C0 are in order from the front.

On the other hand, when the size of the PU to be processed is smaller than the predetermined size, the CU image is a fine image, and the correlation between the motion vectors of the base image and the enhancement image is relatively weak. Therefore, in this case, the registration order of the motion vectors of the block ColBL is determined last. That is, the prediction vector identification information of the motion vector of the block ColBL is the maximum value. The order of registration before the motion vector of the block ColBL is the same as in the conventional case.

For example, when the predetermined size is half the size of the CU, PU171, PU172-1, PU172-2, PU173-1, PU173-2, PU175-1, PU176-2, and PU177-1 in FIG. , And the registration order of the motion vector of the block ColBL in the list for the PU 178-2 is first.

Also, the motion vector registration order of the block ColBL in the list for PUs 174-1 to 174-4, PU 175-2, PU 176-1, PU 177-2, and PU 178-1 in FIG. 10 is the last.

Note that the registration order of the motion vectors of the block ColBL may not be the first or last. That is, if the registration order is earlier than the registration order of the space surrounding CU or the time surrounding CU, it may be the second or the like instead of the top, and if the registration order is later than the registration order of the space surrounding CU or the time surrounding CU, It may be the second from the end, not the last.

On the other hand, when it is determined that the motion vector of the base image is not a candidate for the prediction vector, the list registration unit 133 sequentially starts from the front of the block A0 or the block A1 as in the conventional case illustrated in A of FIG. The registration order is determined so as to be a motion vector, a motion vector of any of blocks B0 to B2, and a motion vector of block H or block C0.

(Description of processing of encoding device)
FIG. 12 is a flowchart illustrating the generation process of the encoding device 10 of FIG.

12, the base encoding unit 11 of the encoding device 10 encodes a base image input from the outside using the HEVC method. In addition, the base encoding unit 11 supplies a motion vector used when the base image is inter-encoded to the enhancement encoding unit 12. Furthermore, the base encoding unit 11 generates an encoded stream including encoded data obtained as a result of encoding, SPS, PPS, and the like as a base stream, and supplies the encoded stream to the synthesizing unit 13.

In step S11, the setting unit 21 (FIG. 5) of the enhancement encoding unit 12 sets SPS. In step S12, the setting unit 21 sets the PPS. The setting unit 21 supplies a parameter set such as SPS or PPS to the encoding unit 22.

In step S13, the encoding unit 22 performs an encoding process that encodes an enhancement image input from the outside using a motion vector supplied from the base encoding unit 11 in a method according to the HEVC method. Details of this encoding process will be described with reference to FIG.

In step S14, the synthesizing unit 13 synthesizes the base stream supplied from the base encoding unit 11 and the enhancement stream supplied from the enhancement encoding unit 12, and generates an encoded stream of all layers. The synthesizing unit 13 supplies the encoded stream of all layers to the transmission unit 14.

In step S15, the transmission unit 14 transmits the encoded stream of all layers supplied from the synthesis unit 13 to a decoding device to be described later, and ends the process.

FIG. 13 is a flowchart illustrating details of the encoding process in step S13 of FIG.

In step S100 of FIG. 13, the A / D conversion unit 101 (FIG. 7) of the encoding unit 22 performs A / D conversion on the input enhancement image, and the screen rearrangement buffer 102 converts the image that is the converted digital data. To supply and memorize.

In step S101, the screen rearrangement buffer 102 rearranges the images of the frames in the stored display order into the frame order for encoding in accordance with the GOP. The screen rearrangement buffer 102 also supplies the image with the rearranged frame order to the arithmetic unit 103, the intra prediction unit 114, and the motion prediction / compensation unit 115.

In step S102, the intra prediction unit 114 performs intra prediction using the PU as a processing unit using the peripheral image supplied from the decoded picture buffer 112 via the selection unit 113. The intra prediction unit 114 performs this intra prediction in all candidate intra prediction modes.

In addition, the intra prediction unit 114 calculates the cost function value of each prediction image using the prediction image generated by the intra prediction of all candidate intra prediction modes and the input image supplied from the screen rearrangement buffer 102. Evaluate and select the optimal intra prediction mode. Then, the intra prediction unit 114 supplies the predicted image and the cost function value generated in the optimal intra prediction mode to the predicted image selection unit 116.

In step S103, the processing unit 131 (FIG. 8) of the motion prediction / compensation unit 115 uses the decoded image read as the reference image from the decoded picture buffer 112 and the input image from the screen rearrangement buffer 102, Inter prediction is performed using the PU as a processing unit. Then, the processing unit 131 performs compensation processing according to the motion vector detected as a result of inter prediction, and generates a predicted image. The processing unit 131 performs such inter prediction and compensation processing in all candidate inter prediction modes.

In addition, the processing unit 131 evaluates the cost function value of each prediction image using the prediction images and input images of all candidate inter prediction modes, and selects an optimal inter prediction mode. Then, the processing unit 131 supplies the predicted image and the cost function value generated in the optimal inter prediction mode to the predicted image selection unit 116.

In step S104, the predicted image selection unit 116, based on the cost function values supplied from the intra prediction unit 114 and the processing unit 131, serves as the source of the predicted image supplied to the calculation unit 103 and the calculation unit 110. Determine whether to select.

When it is determined in step S104 that the intra prediction unit 114 is selected, the prediction image selection unit 116 supplies the prediction image supplied from the intra prediction unit 114 to the calculation unit 103 and the calculation unit 110, and selects the intra prediction unit 114. To be notified. In step S105, the intra prediction unit 114 supplies the intra prediction information to the lossless encoding unit 106, and the process proceeds to step S108.

On the other hand, when it is determined in step S104 that the intra prediction unit is not selected, the predicted image selection unit 116 supplies the predicted image supplied from the processing unit 131 to the calculation unit 103 and the calculation unit 110, and selects the processing unit 131. To be notified.

Accordingly, the processing unit 131 supplies the PU size, the prediction direction, the motion vector, and the reference image specifying information corresponding to the prediction image generated in the optimal inter prediction mode to the prediction information buffer 132 as the prediction information, and stores it. Let Further, the processing unit 131 supplies the motion vector to the vector encoding unit 135, and the optimal inter prediction mode, prediction direction, and reference image specifying information indicating the PU size are supplied to the lossless encoding unit 106 as inter prediction information. Supply.

In step S106, the motion prediction / compensation unit 115 performs list generation processing for generating a list of motion vector prediction vectors corresponding to the predicted image generated in the optimal inter prediction mode. Details of this list generation processing will be described with reference to FIG.

In step S107, the vector encoding unit 135 determines a prediction vector by comparing the prediction vector candidate with the motion vector from the processing unit 131, and generates a difference between the prediction vector and the motion vector as motion vector information. Thus, the motion vector is encoded. The prediction vector candidate is a motion vector identified by the motion vector identification information registered in the list supplied from the list registration unit 133, and is read from the prediction information buffer 132 and the vector buffer 134. After encoding the motion vector, the process proceeds to step S108.

In step S108, the calculation unit 103 calculates a difference between the image read from the screen rearrangement buffer 102 and the predicted image supplied from the intra prediction unit 114 or the motion prediction / compensation unit 115 via the predicted image selection unit 116. Calculate. The calculation unit 103 outputs the difference information obtained as a result to the orthogonal transform unit 104.

In step S109, the orthogonal transform unit 104 performs orthogonal transform on the difference information supplied from the calculation unit 103, and supplies the transform coefficient obtained as a result to the quantization unit 105.

In step S110, the quantization unit 105 quantizes the transform coefficient supplied from the orthogonal transform unit 104, and supplies the quantized transform coefficient to the lossless encoding unit 106 and the inverse quantization unit 108.

In step S111, the inverse quantization unit 108 inversely quantizes the quantized transform coefficient supplied from the quantization unit 105 by a method corresponding to the quantization by the quantization unit 105, and inversely orthogonalizes the obtained transform coefficient. This is supplied to the conversion unit 109.

In step S112, the inverse orthogonal transform unit 109 performs inverse orthogonal transform on the transform coefficient supplied from the inverse quantization unit 108 by a method corresponding to the orthogonal transform processing by the orthogonal transform unit 104, and locally restores the resultant result. The obtained difference information is supplied to the calculation unit 110.

In step S113, the arithmetic unit 110 is supplied with the locally restored difference information supplied from the inverse orthogonal transform unit 109 from the intra prediction unit 114 or the motion prediction / compensation unit 115 via the predicted image selection unit 116. To obtain a reconstructed image. The reconstructed image is supplied to the filter 111 or the decoded picture buffer 112.

In step S114, the filter 111 appropriately performs a filtering process on the reconstructed image supplied from the calculation unit 110, and supplies the decoded image obtained as a result to the decoded picture buffer 112.

In step S115, the decoded picture buffer 112 stores the reconstructed image and the decoded image supplied from the calculation unit 110.

The reconstructed image stored in the decoded picture buffer 112 is read via the selection unit 113 and supplied to the intra prediction unit 114 as a peripheral image when intra coding is performed. In addition, when inter coding is performed, the decoded image stored in the decoded picture buffer 112 is read via the selection unit 113 and supplied to the motion prediction / compensation unit 115 as a reference image.

In step S116, the lossless encoding unit 106 losslessly encodes the quantized transform coefficient from the quantization unit 105 to obtain encoded data. In addition, the lossless encoding unit 106 losslessly encodes the intra prediction information from the intra prediction unit 114 or the inter prediction information from the motion prediction / compensation unit 115, the filter coefficient used in the filter 111, and the like. As part of the information. The lossless encoding unit 106 supplies encoded data in which header information is multiplexed to the accumulation buffer 107.

In step S117, the accumulation buffer 107 temporarily accumulates the encoded data in which the header information supplied from the lossless encoding unit 106 is multiplexed. And a process returns to step S13 of FIG. 12, and progresses to step S14.

FIG. 14 is a flowchart for explaining the details of the list generation processing in step S106 of FIG.

14, the vector buffer 134 stores the motion vector supplied from the base encoding unit 11. In step S132, the list registration unit 133 determines whether the base vector flag included in the SPS supplied from the setting unit 21 in FIG.

If it is determined in step S132 that the base vector flag is 1, the list registration unit 133 determines that the motion vector of the base image is a prediction vector candidate, and the process proceeds to step S133. In step S133, the list registration unit 133 determines the registration order so that the motion vector registration order of the block ColBL comes first.

In step S134, the list registration unit 133 of the motion prediction / compensation unit 115 determines whether or not the block ColBL exists. When it is determined in step S134 that the block ColBL is present, in step S135, the list registration unit 133 determines whether the optimal prediction mode of the block ColBL is the intra prediction mode.

If it is determined in step S135 that the optimal prediction mode of the block ColBL is the intra prediction mode, that is, if there is a motion vector of the block ColBL, the process proceeds to step S136.

In step S136, the list registration unit 133 determines whether the size of the processing target PU held in the prediction information buffer 132 is greater than or equal to N × N (N is an arbitrary positive integer) pixels. If it is determined in step S136 that the size of the PU to be processed is N × N pixels or more, the process proceeds to step S137.

In step S137, based on the registration order determined in step S133, the list registration unit 133 sets the motion vector of the block ColBL as a prediction vector candidate and registers the motion vector identification information of the motion vector in the list. Then, the process proceeds to step S141.

On the other hand, if it is determined in step S136 that the size of the PU to be processed is not greater than N × N pixels, in step S138, the list registration unit 133 changes the registration order so that the registration order of the block ColBL is last. To do.

In step S139, the list registration unit 133 holds the motion vector identification information of the motion vector of the block ColBL based on the registration order changed in step S138, and the process proceeds to step S141.

If it is determined in step S134 that the block ColBL does not exist or the optimal prediction mode of the block ColBL is not the intra prediction mode in step S135, the process proceeds to step S141.

On the other hand, if it is determined in step S132 that the base vector flag is not 1, that is, if the base vector flag is 0, the list registration unit 133 determines that the motion vector of the base image is not a candidate for a prediction vector, The process proceeds to step S140.

In step S140, the list registration unit 133 determines the registration order of motion vectors of blocks other than the block ColBL, that is, the spatial peripheral PU and the temporal peripheral PU, and the process proceeds to step S141.

In step S141, based on the registration order, the list registration unit 133 sets the motion vector of the spatial surrounding PU as a prediction vector candidate, and registers the motion vector identification information of the motion vector in the list.

Specifically, the list registration unit 133 reads the block A0, the block A1, and the blocks B0 to B2 and the prediction direction of the processing target PU and the reference image specifying information from the prediction information buffer 132. The list registration unit 133 sequentially determines whether or not the motion vectors of the block A0 and the block A1 of VEC1 exist based on the read prediction direction and the reference image specifying information, and then determines the determination as VEC2, VEC3. , VEC4 is sequentially performed until it is determined that it exists. Then, the list registration unit 133 uses the motion vector of the existing block A0 or block A1 as a prediction vector candidate, and registers the motion vector identification information of the motion vector in the list.

Further, the list registration unit 133 sequentially determines whether or not motion vectors of the blocks B0 to B2 of the VEC1 exist based on the read prediction direction and the reference image specifying information, and then determines the determination as VEC2, The processing performed in order for VEC3 and VEC4 is performed until it is determined that it exists. Then, the list registration unit 133 sets any one motion vector of the existing blocks B0 to B2 as a prediction vector candidate, and registers motion vector identification information of the motion vector in the list.

In step S142, based on the registration order, the list registration unit 133 sets the motion vector of the temporal peripheral PU as a prediction vector candidate, and registers the motion vector identification information of the motion vector in the list.

Specifically, when there is a motion vector of block H, list registration unit 133 uses the motion vector of block H as a candidate for a prediction vector, and registers motion vector identification information of the motion vector in the list. On the other hand, when the motion vector of block H does not exist, the list registration unit 133 sets the motion vector of block C0 as a candidate for a prediction vector, and registers motion vector identification information of the motion vector in the list.

In step S143, the list registration unit 133 determines whether or not the motion vector identification information of the motion vector of the block ColBL is held, that is, whether or not the process of step S139 has been performed. If it is determined in step S143 that the motion vector identification information of the motion vector of the block ColBL is held, the process proceeds to step S144.

In step S144, the list registration unit 133 registers the motion vector identification information of the motion vector of the block ColBL that is held in the list, and the process proceeds to step S145.

On the other hand, when it is determined in step S143 that the motion vector identification information of the motion vector of the block ColBL is not held, the process proceeds to step S145.

In step S145, the list registration unit 133 determines whether the number of registered motion vector identification information is less than a predetermined number. If it is determined in step S145 that the number of motion vector identification information registered is less than the predetermined number, in step S146, the list registration unit 133 registers motion vector identification information of a motion vector that is 0 in the list. Then, the list registration unit 133 supplies the list to the vector encoding unit 135, returns the process to step S106 in FIG. 13, and proceeds to step S107.

On the other hand, if it is determined in step S145 that the number of motion vector identification information registered is not less than the predetermined number, the list registration unit 133 supplies the list to the vector encoding unit 135. And a process returns to step S106 of FIG. 13, and progresses to step S107.

As described above, the encoding apparatus 10 registers motion vector prediction vector candidates of the enhancement image including the motion vector of the base image in the list based on the PU size as the feature of the enhancement image.

Therefore, for example, when the size of the PU is equal to or larger than a predetermined size, that is, when the motion vector of the base image is highly likely to be a prediction vector, the motion vector of the base image is set before the motion vector of the enhancement image. You can register to the list. As a result, the information amount of prediction vector identification information can be reduced and encoding efficiency can be improved.

In addition, for example, when the size of the PU is smaller than a predetermined size, that is, when it is unlikely that the motion vector of the base image is the prediction vector, the enhancement image motion vector is listed before the motion vector of the base image. Can be registered. As a result, the information amount of prediction vector identification information can be reduced and encoding efficiency can be improved.

(Configuration example of one embodiment of decoding device)
FIG. 15 is a block diagram illustrating a configuration example of an embodiment of a decoding device to which the present technology is applied, which decodes an encoded stream of all layers transmitted from the encoding device 10 of FIG.

15 includes a receiving unit 181, a separating unit 182, a base decoding unit 183, and an enhancement decoding unit 184.

The receiving unit 181 receives the encoded stream of all layers transmitted from the encoding device 10 in FIG. 4 and supplies it to the separating unit 182.

The separating unit 182 separates the base stream from the encoded streams of all layers and supplies the base stream to the base decoding unit 183, and separates the enhancement stream and supplies it to the enhancement decoding unit 184.

The base decoding unit 183 is configured in the same manner as a conventional HEVC decoding device, decodes the base stream supplied from the separation unit 182 using the HEVC method, and generates a base image. However, the base decoding unit 183 supplies the motion vector used at the time of inter decoding of the base image to the enhancement decoding unit 184. The base decoding unit 183 outputs the generated base image.

The enhancement decoding unit 184 decodes the enhancement stream supplied from the separation unit 182 by a method according to the HEVC method, and generates an enhancement image. At this time, the enhancement decoding unit 184 refers to the motion vector supplied from the base decoding unit 183. The enhancement decoding unit 184 outputs the generated enhancement image.

(Configuration example of enhancement decoding unit)
FIG. 16 is a block diagram illustrating a configuration example of the enhancement decoding unit 184 of FIG.

The enhancement decoding unit 184 in FIG. 16 includes an extraction unit 201 and a decoding unit 202.

The extraction unit 201 of the enhancement decoding unit 184 extracts SPS, PPS, encoded data, and the like from the enhancement stream supplied from the separation unit 182 in FIG.

The decoding unit 202 refers to the motion vector of the base image supplied from the base decoding unit 183 in FIG. 15, and decodes the encoded data supplied from the extraction unit 201 by a method according to the HEVC method. At this time, the decoding unit 202 also refers to SPS, PPS, and the like supplied from the extraction unit 201 as necessary. The decoding unit 202 outputs an image obtained as a result of decoding as an enhancement image.

(Configuration example of decoding unit)
FIG. 17 is a block diagram illustrating a configuration example of the decoding unit 202 of FIG.

17 includes an accumulation buffer 301, a lossless decoding unit 302, an inverse quantization unit 303, an inverse orthogonal transform unit 304, a calculation unit 305, a loop filter 306, a screen rearrangement buffer 307, and a D / A conversion unit 308. Have In addition, the decoding unit 202 includes a decoded picture buffer 309, a selection unit 310, an intra prediction unit 311, a motion compensation unit 312, and a selection unit 313. The decoding unit 202 decodes the encoded data of the enhancement image supplied from the extraction unit 201 in FIG.

The accumulation buffer 301 accumulates the encoded data of the enhancement image supplied from the extraction unit 201 and supplies the encoded data to the lossless decoding unit 302 at a predetermined timing. The lossless decoding unit 302 performs lossless decoding of the encoded data supplied from the accumulation buffer 301 by a method corresponding to the encoding method of the lossless encoding unit 106 in FIG. The lossless decoding unit 302 supplies the quantized transform coefficient obtained by the lossless decoding to the inverse quantization unit 303.

Further, the lossless decoding unit 302 supplies the intra prediction information to the intra prediction unit 311 when the intra prediction information is obtained by decoding the encoded data, and when the inter prediction information is obtained, the inter prediction information is obtained. The prediction information is supplied to the motion compensation unit 312.

The inverse quantization unit 303 inversely quantizes the quantized transform coefficient supplied from the lossless decoding unit 302 by a method corresponding to the quantization method of the quantization unit 105 in FIG. 7, and reverses the obtained transform coefficient. This is supplied to the orthogonal transform unit 304. The inverse orthogonal transform unit 304 performs inverse orthogonal transform on the transform coefficient supplied from the inverse quantization unit 303 by a method corresponding to the orthogonal transform method of the orthogonal transform unit 104 in FIG. 7 to obtain difference information.

The difference information obtained by the inverse orthogonal transform is supplied to the calculation unit 305. In addition, a prediction image is supplied from the intra prediction unit 311 or the motion compensation unit 312 to the calculation unit 305 via the selection unit 313.

The calculation unit 305 adds the difference information and the predicted image to obtain a reconstructed image. The arithmetic unit 305 supplies the reconstructed image to the loop filter 306 or the decoded picture buffer 309.

The loop filter 306 performs a filtering process on the reconstructed image supplied from the calculation unit 305 as appropriate, similarly to the filter 111 in FIG. 7, and generates a decoded image. Note that when a filter coefficient is obtained by the lossless decoding by the lossless decoding unit 302, the loop filter 306 performs a filter process using the filter coefficient. The loop filter 306 supplies the decoded image to the screen rearrangement buffer 307 and the decoded picture buffer 309.

The screen rearrangement buffer 307 rearranges the decoded images supplied from the loop filter 306. That is, the order of frames rearranged for the encoding order by the screen rearrangement buffer 102 in FIG. 7 is rearranged in the original display order. The D / A conversion unit 308 D / A converts the decoded image supplied from the screen rearrangement buffer 307. The D / A conversion unit 308 outputs the enhancement image obtained as a result.

The decoded picture buffer 309 stores the reconstructed image supplied from the calculation unit 305. The decoded picture buffer 309 stores the decoded image supplied from the loop filter 306.

The decoded picture buffer 309 supplies the stored reconstructed image to the intra prediction unit 311 via the selection unit 310 at a predetermined timing or based on an external request from the intra prediction unit 311 or the like. . The decoded picture buffer 309 also stores the decoded image stored at a predetermined timing or based on a request from the outside such as the motion parallax prediction / compensation unit 115 via the selection unit 310. To supply.

The intra prediction unit 311 performs basically the same processing as the intra prediction unit 114 in FIG. However, the intra prediction unit 311 performs intra prediction in the optimal intra prediction mode of the intra prediction information supplied from the lossless decoding unit 302 only for a region where a prediction image is generated by intra prediction at the time of encoding. . The intra prediction unit 311 supplies a prediction image obtained as a result of the intra prediction to the selection unit 313.

The motion compensation unit 312 reads the decoded image specified by the reference image specifying information included in the inter prediction information supplied from the lossless decoding unit 302 from the decoded picture buffer 309 as a reference image.

In addition, the motion compensation unit 312 is based on the motion vector from the base decoding unit 183 and the base vector flag included in the SPS from the extraction unit 201, similarly to the motion parallax prediction / compensation unit 115 in FIG. A list in which motion vector identification information of motion vectors that are candidates for the above is registered is generated. The motion compensation unit 312 determines a motion vector identified by the motion vector identification information identified by the prediction vector identification information included in the inter prediction information from the generated list as a prediction vector. Then, the motion compensation unit 312 decodes the motion vector of the processing target PU by adding the prediction vector and the motion vector information included in the inter prediction information.

The motion compensation unit 312 performs a compensation process for the optimal inter prediction mode included in the inter prediction information, using the PU as a processing unit, based on the generated motion vector and the reference image, and generates a predicted image. Based on the inter prediction information supplied from the lossless decoding unit 302, the motion compensation unit 312 performs this compensation processing only on a region where inter prediction has been performed at the time of encoding. The motion compensation unit 312 supplies the generated predicted image to the calculation unit 305 via the selection unit 313.

The selection unit 313 supplies the prediction image supplied from the intra prediction unit 311 or the prediction image supplied from the motion compensation unit 312 to the calculation unit 305.

(Configuration example of motion compensation unit)
FIG. 18 is a block diagram illustrating a configuration example of the motion compensation unit 312 in FIG.

18 includes a processing unit 331, a prediction information buffer 332, a list registration unit 333, a vector buffer 334, and a vector decoding unit 335.

The processing unit 331 of the motion compensation unit 312 refers to the decoded image stored in the decoded picture buffer 309 based on the prediction direction and the reference image specifying information included in the inter prediction information from the lossless decoding unit 302 in FIG. Read as an image. The processing unit 331 uses the motion vector supplied from the vector decoding unit 335 and the reference image to perform compensation processing in an optimal inter prediction mode, and generates a predicted image. The processing unit 331 supplies the predicted image to the selection unit 313 in FIG.

The prediction information buffer 332 stores, as prediction information, the PU size, prediction direction, motion vector, and reference image specifying information represented by the optimal inter prediction mode, included in the inter prediction information supplied from the lossless decoding unit 302. .

The list registration unit 333 determines whether or not to use the motion vector of the base image as a prediction vector candidate based on the base vector flag included in the SPS supplied from the extraction unit 201 of FIG.

When the list registration unit 333 sets the motion vector of the base image as a prediction vector candidate, the PU size, prediction direction, and reference image stored in the prediction information buffer 332 are the same as the list registration unit 133 in FIG. Based on the specific information, the motion vector identification information is registered in the list.

On the other hand, when the motion vector of the base image is not used as a candidate for a prediction vector, the list registration unit 333 is based on the prediction direction and the reference image specifying information stored in the prediction information buffer 332 as in the list registration unit 133. Register motion vector identification information in the list. Then, the list registration unit 333 supplies the generated list to the vector decoding unit 335. Note that the motion vector identification information in the list is given prediction vector identification information in the order of registration.

The vector buffer 334 stores the motion vector supplied from the base decoding unit 183.

The vector decoding unit 335 acquires motion vector identification information identified by the prediction vector identification information included in the inter prediction information supplied from the lossless decoding unit 302 from the list supplied from the list registration unit 333. Then, the vector decoding unit 335 reads the motion vector identified by the motion vector identification information from the prediction information buffer 332 or the vector buffer 334 as a prediction vector.

The vector decoding unit 335 performs motion vector decoding by adding the motion vector information and the prediction vector included in the inter prediction information supplied from the lossless decoding unit 302. The vector decoding unit 335 supplies the motion vector to the processing unit 331.

(Description of processing of decoding device)
FIG. 19 is a flowchart illustrating image generation processing of the decoding device 180 of FIG.

19, the reception unit 181 of the decoding device 180 receives the encoded stream of all layers transmitted from the encoding device 10 of FIG. 4 and supplies the encoded stream to the separation unit 182. In step S171, the separation unit 182 separates the base stream and the enhancement stream from the encoded streams of all layers supplied from the reception unit 181 and supplies the separated base stream and enhancement stream to the enhancement decoding unit 184.

In step S172, the base decoding unit 183 decodes the base stream supplied from the separation unit 182 using the HEVC method, and generates a base image. In addition, the base decoding unit 183 supplies the motion vector used at the time of inter decoding of the base image to the enhancement decoding unit 184. The base decoding unit 183 outputs the generated base image.

In step S173, the extraction unit 201 (FIG. 16) of the enhancement decoding unit 184 extracts the SPS from the enhancement stream supplied from the separation unit 182 and supplies the SPS to the decoding unit 202. In step S174, the extraction unit 201 extracts PPS from the enhancement stream and supplies the PPS to the decoding unit 202.

In step S175, the extraction unit 201 extracts encoded data from the enhancement stream and supplies the encoded data to the decoding unit 202. In step S176, the decoding unit 202 uses the motion vector supplied from the base decoding unit 183 to perform decoding processing that decodes the encoded data supplied from the extraction unit 201 in a method according to the HEVC method, and the processing ends. To do. Details of the decoding process will be described with reference to FIG.

FIG. 20 is a flowchart for explaining the details of the decoding process in step S176 of FIG.

In step S301 in FIG. 20, the accumulation buffer 301 accumulates the encoded data of the enhancement image supplied from the extraction unit 201 in FIG. 16, and supplies the encoded data to the lossless decoding unit 302 at a predetermined timing.

In step S302, the lossless decoding unit 302 performs lossless decoding of the encoded data supplied from the accumulation buffer 301 by a method corresponding to the encoding method of the lossless encoding unit 106 in FIG. The lossless decoding unit 302 supplies the quantized transform coefficient obtained by the lossless decoding to the inverse quantization unit 303.

In step S303, the inverse quantization unit 303 inversely quantizes the quantized transform coefficient supplied from the lossless decoding unit 302 by a method corresponding to the quantization method of the quantization unit 105 in FIG. The transform coefficient is supplied to the inverse orthogonal transform unit 304. In step S304, the inverse orthogonal transform unit 304 performs inverse orthogonal transform on the transform coefficient supplied from the inverse quantization unit 303 by a method corresponding to the orthogonal transform method of the orthogonal transform unit 104 in FIG. 7 to obtain difference information. The difference information obtained by the inverse orthogonal transform is supplied to the calculation unit 305.

In step S305, the intra prediction unit 311 determines whether or not intra prediction information is supplied from the lossless decoding unit 302. When it is determined in step S305 that intra prediction information has been supplied, in step S306, the intra prediction unit 311 performs intra prediction basically similar to the intra prediction unit 114 in FIG. The intra prediction unit 311 supplies the prediction image obtained as a result of the intra prediction to the selection unit 313, and the process proceeds to step S310.

On the other hand, if it is determined in step S305 that intra prediction information is not supplied, that is, if the motion compensation unit 312 acquires inter prediction information, the process proceeds to step S307.

In step S307, the motion compensation unit 312 performs a list generation process similar to the list generation process of FIG. In step S308, the vector decoding unit 335 (FIG. 18) of the motion compensation unit 312 is based on the list generated by the process of step S307 and the prediction vector identification information included in the inter prediction information from the lossless decoding unit 302. Then, the motion vector information included in the inter prediction information is decoded by the AMVP method. And the vector decoding part 335 supplies the motion vector obtained as a result to the process part 331, and advances a process to step S309.

In step S309, the processing unit 331 uses the PU as a processing unit based on the motion vector from the vector decoding unit 335 and the reference image read from the decoded picture buffer 309, and performs the optimal inter prediction included in the inter prediction information. A mode compensation process is performed to generate a predicted image. The processing unit 331 supplies the generated predicted image to the calculation unit 305 via the selection unit 313, and advances the processing to step S310.

In step S310, the calculation unit 305 adds the difference information supplied from the inverse orthogonal transform unit 304 and the predicted image supplied from the selection unit 313 to obtain a reconstructed image. The arithmetic unit 305 supplies the reconstructed image to the loop filter 306 or the decoded picture buffer 309.

In step S311, the loop filter 306 generates a decoded image by appropriately performing filter processing on the reconstructed image supplied from the calculation unit 305, similarly to the filter 111 in FIG. Note that when a filter coefficient is obtained by the lossless decoding by the lossless decoding unit 302, the loop filter 306 performs a filter process using the filter coefficient. The loop filter 306 supplies the decoded image as the filter processing result to the screen rearrangement buffer 307 and the decoded picture buffer 309.

In step S312, the screen rearrangement buffer 307 rearranges the decoded images supplied from the loop filter 306. In step S313, the D / A conversion unit 308 D / A converts the decoded image supplied from the screen rearrangement buffer 307, and outputs the enhancement image obtained as a result.

In step S314, the decoded picture buffer 309 stores the reconstructed image supplied from the calculation unit 305 and the decoded image supplied from the loop filter 306.

The reconstructed image stored in the decoded picture buffer 309 is supplied to the intra prediction unit 311 via the selection unit 310. The decoded image stored in the decoded picture buffer 309 is supplied to the motion compensation unit 312 via the selection unit 310. After the process of step S314, the process returns to step S176 of FIG. 19 and ends.

As described above, the decoding device 180 registers motion vector prediction vector candidates of the enhancement image including the motion vector of the base image in the list based on the PU size as the feature of the enhancement image. Therefore, it is possible to decode the enhancement stream transmitted from the encoding device 10 and improved in encoding efficiency.

In the above description, the registration order of the motion vector of the block ColBL is determined at the head or the end based on the size of the PU to be processed. However, whether or not the motion vector of the block ColBL is registered is determined. May be. This case will be described with reference to FIGS. 21 and 22 below.

(Other methods for determining prediction vector candidate registration order)
FIG. 21 is a diagram illustrating a method for determining the registration order of prediction vector candidates in this case.

As shown in FIG. 21, when it is determined that the motion vector of the base image is a prediction vector candidate, when the size of the PU to be processed is equal to or larger than a predetermined size, the registration order is the same as in the case described with reference to FIG. Is determined. That is, the registration order of the motion vectors of the block ColBL is determined at the top.

On the other hand, when the size of the PU to be processed is smaller than the predetermined size, the motion vector of the block ColBL is not regarded as a prediction vector candidate. Therefore, as in the conventional case shown in FIG. 2A, the motion vector of the block A0 or the block A1, the motion vector of any of the blocks B0 to B2, and the motion vector of the block H or the block C0 are sequentially from the front. The registration order is determined as follows.

Also, the motion vectors of the block ColBL for the PUs 174-1 to 174-4, PU175-2, PU176-1, PU177-2, and PU178-1 in FIG. 10 are not registered in the list.

On the other hand, when it is determined that the motion vector of the base image is not a candidate for the prediction vector, the registration order is determined as in the case described with reference to FIG.

As described above, in the determination method of FIG. 21, when the size of the PU to be processed is smaller than a predetermined size, that is, when the correlation between the motion vectors of the base image and the enhancement image is relatively weak, the motion vectors of the base image are listed. Not registered. Therefore, it is possible to reduce the prediction vector identification information of the motion vector of the enhancement image that is highly likely to be a prediction vector, and to improve the encoding efficiency.

(Description of other list generation processing)
FIG. 22 is a flowchart for explaining the list generation processing by the list registration unit 133 when the registration order is determined by the determination method of FIG.

22 is the same as the processing of steps S131 to S137 of FIG. 14, and thus the description thereof is omitted.

If it is determined in step S332 that the base vector flag is not 1, the list registration unit 133 determines that the motion vector of the base image is not a prediction vector candidate, and the process proceeds to step S338. If it is determined in step S336 that the size of the PU to be processed is not greater than or equal to N × N pixels, the process proceeds to step S338.

In step S338, the list registration unit 133 determines the registration order of motion vectors of blocks other than the block ColBL, and advances the process to step S339.

Since the processes in steps S339 to S342 are the same as the processes in steps S141, S142, S145, and S146 in FIG.

As described above, in the list generation process in FIG. 22, the processes in steps S138 and S139 and the processes in steps S143 and S144 in FIG. 14 are not performed, so that the list can be easily generated. 22 is performed by the list registration unit 133, the list generation unit 333 performs the same list generation process as the list generation process of FIG.

In the above description, the registration order is determined based on the PU size of the enhancement image. However, the registration order may be determined based on the PU size of the base image corresponding to the PU. Also, the registration order may be determined based on the sizes of both the enhancement image PU and the corresponding base image PU.

In this case, for example, when the size of at least one of the enhancement image PU and the corresponding base image PU is smaller than N × N pixels, the motion vector of the block ColBL is registered at the end of the list or not registered. . On the other hand, when both sizes are large, the motion vector of the block ColBL is registered at the top of the list.

Also, the registration order may be determined based on the CU size instead of the PU size. In this case, for example, as shown in FIG. 23, when the size of the CU is 32 × 32 pixels or more, the motion vector of the block ColBL is registered at the top of the list. On the other hand, when the size of the CU is 16 × 16 pixels or 8 × 8 pixels smaller than 32 × 32 pixels, the motion vector of the block ColBL is registered at the end of the list or not registered.

Note that the size of the CU used for determining the registration order is the size of at least one of the CU to be processed for the enhancement image and the CU of the base image corresponding to the CU, as in the case of the PU.

Furthermore, the registration order may be determined based on the sizes of both the CU and the PU. In this case, for example, when the size of the CU is 32 × 32 pixels or more, the motion vector of the block ColBL is registered at the top of the list. If the CU size is 16 × 16 pixels or 8 × 8 pixels smaller than 32 × 32 pixels, and the PU size is N × N pixels or more, the motion vector of the block ColBL is registered at the top of the list. The

On the other hand, if the CU size is 16x16 pixels or 8x8 pixels smaller than 32x32 pixels and the PU size is smaller than NxN pixels, is the motion vector of the block ColBL registered at the end of the list? Or not registered.

Also, the registration order may be determined based on the TU size, not the PU or CU size.

Furthermore, the registration order may be determined based on the type of scalability function instead of the PU size. In this case, for example, as shown in FIG. 24, when the type of the scalability function is spatial scalability where the division value α obtained by dividing the spatial resolution of the enhancement image (EL) by the spatial resolution of the base image (BL) is greater than 1. The motion vector of block ColBL is registered at the end of the list or not registered.

That is, when the division value α is larger than 1, it is necessary to scale the motion vector in order to make the motion vector of the base image a candidate for a prediction vector. For example, when the spatial resolution of the enhancement image is twice the spatial resolution of the base image, it is necessary to scale the motion vector of the base image to twice. Therefore, in this case, the correlation between the motion vectors of the base image and the enhancement image is relatively weak. Therefore, the motion vector of the block ColBL of the base image is registered at the end of the list or not registered.

On the other hand, when the type of the scalability function is scalability with the division value α being 1, the motion vector need not be scaled, so the motion vector of the block ColBL is registered at the top of the list.

Further, the registration order of the list may be determined based on the quantization parameter of the base image and the enhancement image instead of the PU size. In this case, for example, as shown in FIG. 25, when the quantization parameter QP _E of the enhancement image is larger than the quantization parameter QP _B of the base image, the motion vector of the block ColBL is registered at the top of the list. On the other hand, when the quantization parameter QP _E is equal to or less than the quantization parameter QP _B , the motion vector of the block ColBL is registered at the end of the list or not registered.

As the quantization parameter QP _E (QP _B ), a quantization parameter in units of pictures (PicQP) and a quantization parameter in units of slices (Slice QP) can be used in addition to the quantization parameter in units of CUs. When quantization parameters in units of pictures or slices are used, it is possible to further reduce the information amount of quantization parameters that are held for determining the registration order of the list.

Further, the hierarchy peripheral PU is not limited to the block ColBL, and may be another PU as long as it is the position of the base image whose position in the screen is the same as the PU to be processed or the PU around the position. . In addition, the number of hierarchical peripheral PUs may be plural. For example, as illustrated in FIG. 26, the hierarchical peripheral PU may be a block ColBL and a block HBL that is the lower right PU of the base image region whose position in the screen is the same as the processing target PU.

In this case, when the registration order of the motion vector of the base image is earlier than the motion vector of the temporal peripheral PU, first, the motion vector of the block ColBL or the block HBL is selected as a prediction vector candidate and registered in the list. Then, the motion vector of the block C0 or the block H at the position corresponding to the unregistered motion vector is registered in the list.

Specifically, when the motion vector of the block ColBL is registered in the list, the motion vector of the block H is registered in the list, and when the motion vector of the block HBL is registered in the list, the motion vector of the block C0 is registered in the list. be registered.

On the other hand, when the registration order of the motion vectors of the base image is slower than the motion vectors of the time peripheral PU, first, the motion vector of the block C0 or the block H is selected as a prediction vector candidate and registered in the list. Then, the motion vector of the block ColBL or the block HBL at the position corresponding to the motion vector not registered is registered in the list.

Specifically, when the motion vector of the block C0 is registered in the list, the motion vector of the block HBL is registered in the list. When the motion vector of the block H is registered in the list, the motion vector of the block ColBL is registered in the list. be registered.

Also, the base vector flag may be set not in sequence units but in picture units or sequence units. For example, when the base vector flag is set in units of pictures, the base vector flag (pps_enable_BLMV_flag) in units of pictures is included in the PPS as shown in the 19th line of FIG.

When the base vector flag is set in units of slices, the base vector flag (slice_enable_BLMV_flag) in units of slices is included in the slice header as shown in the sixth line of FIG.

It should be noted that the enhancement stream may include a change flag and a switch flag instead of the base vector flag. The change flag is a flag for identifying a motion vector of a base image as a candidate for a prediction vector and determining that the order of registration of the base image in the motion vector list is determined based on the feature of the image. The switching flag is a flag for identifying whether the motion vector of the base image is a candidate for a prediction vector and switching the presence or absence of registration of the motion vector of the base image in the list based on the feature of the image.

In the above description, the case where the motion vector is encoded and decoded by the AMVP method has been described. Similarly, when the motion vector is encoded and decoded by the Merge method, the prediction vector candidate is also based on the feature of the image. A list of is generated.

<Application to multi-view image coding and multi-view image decoding>
The series of processes described above can be applied to multi-view image encoding / multi-view image decoding. FIG. 29 shows an example of a multi-view image encoding method.

29, the multi-viewpoint image includes a plurality of viewpoint images, and a predetermined one viewpoint image among the plurality of viewpoints is designated as the base view image. Each viewpoint image other than the base view image is treated as a non-base view image. When multi-view image encoding is performed by the Scalability function, the base view image is encoded as a base layer image, and the non-base view image is encoded as an enhancement image.

When performing multi-view image coding as shown in FIG. 29, the quantization parameter difference can be taken in each view (same view):
(1) base-view:
(1-1) dQP (base view) = Current_CU_QP (base view)-LCU_QP (base view)
(1-2) dQP (base view) = Current_CU_QP (base view)-Previsous_CU_QP (base view)
(1-3) dQP (base view) = Current_CU_QP (base view)-Slice_QP (base view)
(2) non-base-view:
(2-1) dQP (non-base view) = Current_CU_QP (non-base view)-LCU_QP (non-base view)
(2-2) dQP (non-base view) = Current QP (non-base view)-Previsous QP (non-base view)
(2-3) dQP (non-base view) = Current_CU_QP (non-base view)-Slice_QP (non-base view)

When performing multi-view image coding, it is also possible to take quantization parameter differences in each view (different views):
(3) base-view / non-base view:
(3-1) dQP (inter-view) = Slice_QP (base view)-Slice_QP (non-base view)
(3-2) dQP (inter-view) = LCU_QP (base view)-LCU_QP (non-base view)
(4) non-base view / non-base view:
(4-1) dQP (inter-view) = Slice_QP (non-base view i) −Slice_QP (non-base view j)
(4-2) dQP (inter-view) = LCU_QP (non-base view i)-LCU_QP (non-base view j)

In this case, the above (1) to (4) can be used in combination. For example, in the non-base view, a method of obtaining a quantization parameter difference at the slice level between the base view and the non-base view (combining 3-1 and 2-3), between the base view and the non-base view The method of taking the difference of the quantization parameter at the LCU level (combining 3-2 and 2-1) can be considered. Thus, by applying the difference repeatedly, the encoding efficiency can be improved even when multi-viewpoint encoding is performed.

Similarly to the method described above, a flag for identifying whether or not there is a dQP whose value is not 0 can be set for each of the above dQPs.

<Other examples of encoding using the Scalability function>
FIG. 30 shows another example of encoding by the scalability function.

As shown in FIG. 30, in the encoding by the scalability function, the difference of the quantization parameter can be taken in each layer (same layer):
(1) base-layer:
(1-1) dQP (base layer) = Current_CU_QP (base layer)-LCU_QP (base layer)
(1-2) dQP (base layer) = Current_CU_QP (base layer)-Previsous_CU_QP (base layer)
(1-3) dQP (base layer) = Current_CU_QP (base layer)-Slice_QP (base layer)
(2) non-base-layer:
(2-1) dQP (non-base layer) = Current_CU_QP (non-base layer)-LCU_QP (non-base layer)
(2-2) dQP (non-base layer) = Current QP (non-base layer)-Previsous QP (non-base layer)
(2-3) dQP (non-base layer) = Current_CU_QP (non-base layer) −Slice_QP (non-base layer)

It is also possible to take quantization parameter differences in each layer (different layers):
(3) base-layer / non-base layer:
(3-1) dQP (inter-layer) = Slice_QP (base layer)-Slice_QP (non-base layer)
(3-2) dQP (inter-layer) = LCU_QP (base layer)-LCU_QP (non-base layer)
(4) non-base layer / non-base layer:
(4-1) dQP (inter-layer) = Slice_QP (non-base layer i) −Slice_QP (non-base layer j)
(4-2) dQP (inter-layer) = LCU_QP (non-base layer i)-LCU_QP (non-base layer j)

In this case, the above (1) to (4) can be used in combination. For example, in the non-base layer, a method of obtaining a difference in quantization parameter at the slice level between the base layer and the non-base layer (combining 3-1 and 2-3), between the base layer and the non-base layer The method of taking the difference of the quantization parameter at the LCU level (combining 3-2 and 2-1) can be considered. In this manner, by applying the difference repeatedly, the encoding efficiency can be improved even when hierarchical encoding is performed.

<Description of computer to which this technology is applied>
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

FIG. 31 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processing by a program.

In the computer, a CPU (Central Processing Unit) 601, a ROM (Read Only Memory) 602, and a RAM (Random Access Memory) 603 are connected to each other via a bus 604.

An input / output interface 605 is further connected to the bus 604. An input unit 606, an output unit 607, a storage unit 608, a communication unit 609, and a drive 610 are connected to the input / output interface 605.

The input unit 606 includes a keyboard, a mouse, a microphone, and the like. The output unit 607 includes a display, a speaker, and the like. The storage unit 608 includes a hard disk, a nonvolatile memory, and the like. The communication unit 609 includes a network interface or the like. The drive 610 drives a removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 601 loads the program stored in the storage unit 608 to the RAM 603 via the input / output interface 605 and the bus 604 and executes the program, for example. Is performed.

The program executed by the computer (CPU 601) can be provided by being recorded on a removable medium 611 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 608 via the input / output interface 605 by attaching the removable medium 611 to the drive 610. Further, the program can be received by the communication unit 609 via a wired or wireless transmission medium and installed in the storage unit 608. In addition, the program can be installed in the ROM 602 or the storage unit 608 in advance.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

<Example configuration of television device>
FIG. 32 illustrates a schematic configuration of a television apparatus to which the present technology is applied. The television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, and an external interface unit 909. Furthermore, the television apparatus 900 includes a control unit 910, a user interface unit 911, and the like.

The tuner 902 selects a desired channel from the broadcast wave signal received by the antenna 901, demodulates it, and outputs the obtained encoded bit stream to the demultiplexer 903.

The demultiplexer 903 extracts video and audio packets of the program to be viewed from the encoded bit stream, and outputs the extracted packet data to the decoder 904. Further, the demultiplexer 903 supplies a packet of data such as EPG (Electronic Program Guide) to the control unit 910. If scrambling is being performed, descrambling is performed by a demultiplexer or the like.

The decoder 904 performs packet decoding processing, and outputs video data generated by the decoding processing to the video signal processing unit 905 and audio data to the audio signal processing unit 907.

The video signal processing unit 905 performs noise removal, video processing according to user settings, and the like on the video data. The video signal processing unit 905 generates video data of a program to be displayed on the display unit 906, image data by processing based on an application supplied via a network, and the like. The video signal processing unit 905 generates video data for displaying a menu screen for selecting an item and the like, and superimposes the video data on the video data of the program. The video signal processing unit 905 generates a drive signal based on the video data generated in this way, and drives the display unit 906.

The display unit 906 drives a display device (for example, a liquid crystal display element or the like) based on a drive signal from the video signal processing unit 905 to display a program video or the like.

The audio signal processing unit 907 performs predetermined processing such as noise removal on the audio data, performs D / A conversion processing and amplification processing on the processed audio data, and outputs the audio data to the speaker 908.

The external interface unit 909 is an interface for connecting to an external device or a network, and transmits and receives data such as video data and audio data.

A user interface unit 911 is connected to the control unit 910. The user interface unit 911 includes an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 910.

The control unit 910 is configured using a CPU (Central Processing Unit), a memory, and the like. The memory stores a program executed by the CPU, various data necessary for the CPU to perform processing, EPG data, data acquired via a network, and the like. The program stored in the memory is read and executed by the CPU at a predetermined timing such as when the television device 900 is activated. The CPU executes each program to control each unit so that the television device 900 operates in accordance with the user operation.

Note that the television device 900 includes a bus 912 for connecting the tuner 902, the demultiplexer 903, the video signal processing unit 905, the audio signal processing unit 907, the external interface unit 909, and the control unit 910.

In the thus configured television apparatus, the decoder 904 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, when the motion vector of the base layer image is used as the motion vector prediction vector candidate of the enhancement layer image, it is possible to decode the encoded stream that improves the encoding efficiency.

<Configuration example of mobile phone>
FIG. 33 illustrates a schematic configuration of a mobile phone to which the present technology is applied. The cellular phone 920 includes a communication unit 922, an audio codec 923, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, and a control unit 931. These are connected to each other via a bus 933.

In addition, an antenna 921 is connected to the communication unit 922, and a speaker 924 and a microphone 925 are connected to the audio codec 923. Further, an operation unit 932 is connected to the control unit 931.

The mobile phone 920 performs various operations such as transmission / reception of voice signals, transmission / reception of e-mail and image data, image shooting, and data recording in various modes such as a voice call mode and a data communication mode.

In the voice call mode, the voice signal generated by the microphone 925 is converted into voice data and compressed by the voice codec 923 and supplied to the communication unit 922. The communication unit 922 performs audio data modulation processing, frequency conversion processing, and the like to generate a transmission signal. The communication unit 922 supplies a transmission signal to the antenna 921 and transmits it to a base station (not shown). In addition, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and supplies the obtained audio data to the audio codec 923. The audio codec 923 performs data expansion of the audio data and conversion to an analog audio signal and outputs the result to the speaker 924.

In the data communication mode, when mail transmission is performed, the control unit 931 receives character data input by operating the operation unit 932 and displays the input characters on the display unit 930. In addition, the control unit 931 generates mail data based on a user instruction or the like in the operation unit 932 and supplies the mail data to the communication unit 922. The communication unit 922 performs mail data modulation processing, frequency conversion processing, and the like, and transmits the obtained transmission signal from the antenna 921. In addition, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and restores mail data. This mail data is supplied to the display unit 930 to display the mail contents.

Note that the mobile phone 920 can also store the received mail data in a storage medium by the recording / playback unit 929. The storage medium is any rewritable storage medium. For example, the storage medium is a removable medium such as a semiconductor memory such as a RAM or a built-in flash memory, a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card.

When transmitting image data in the data communication mode, the image data generated by the camera unit 926 is supplied to the image processing unit 927. The image processing unit 927 performs encoding processing of image data and generates encoded data.

The demultiplexing unit 928 multiplexes the encoded data generated by the image processing unit 927 and the audio data supplied from the audio codec 923 by a predetermined method, and supplies the multiplexed data to the communication unit 922. The communication unit 922 performs modulation processing and frequency conversion processing of multiplexed data, and transmits the obtained transmission signal from the antenna 921. In addition, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and restores multiplexed data. This multiplexed data is supplied to the demultiplexing unit 928. The demultiplexing unit 928 performs demultiplexing of the multiplexed data, and supplies the encoded data to the image processing unit 927 and the audio data to the audio codec 923. The image processing unit 927 performs a decoding process on the encoded data to generate image data. The image data is supplied to the display unit 930 and the received image is displayed. The audio codec 923 converts the audio data into an analog audio signal, supplies the analog audio signal to the speaker 924, and outputs the received audio.

In the cellular phone device configured as described above, the image processing unit 927 is provided with the function of the image processing device (image processing method) of the present application. For this reason, when the motion vector of the base layer image is used as a candidate for the motion vector prediction vector of the enhancement layer image, the encoding efficiency can be improved. In addition, when the motion vector of the base layer image is used as a motion vector prediction vector candidate for the enhancement layer image, it is possible to decode an encoded stream that improves the encoding efficiency.

<Configuration example of recording / reproducing apparatus>
FIG. 34 illustrates a schematic configuration of a recording / reproducing apparatus to which the present technology is applied. The recording / reproducing apparatus 940 records, for example, audio data and video data of a received broadcast program on a recording medium, and provides the recorded data to the user at a timing according to a user instruction. The recording / reproducing device 940 can also acquire audio data and video data from another device, for example, and record them on a recording medium. Further, the recording / reproducing apparatus 940 decodes and outputs the audio data and video data recorded on the recording medium, thereby enabling image display and audio output on the monitor apparatus or the like.

The recording / reproducing apparatus 940 includes a tuner 941, an external interface unit 942, an encoder 943, an HDD (Hard Disk Drive) unit 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) unit 948, a control unit 949, A user interface unit 950 is included.

Tuner 941 selects a desired channel from a broadcast signal received by an antenna (not shown). The tuner 941 outputs an encoded bit stream obtained by demodulating the received signal of a desired channel to the selector 946.

The external interface unit 942 includes at least one of an IEEE 1394 interface, a network interface unit, a USB interface, a flash memory interface, and the like. The external interface unit 942 is an interface for connecting to an external device, a network, a memory card, and the like, and receives data such as video data and audio data to be recorded.

The encoder 943 performs encoding by a predetermined method when the video data and audio data supplied from the external interface unit 942 are not encoded, and outputs an encoded bit stream to the selector 946.

The HDD unit 944 records content data such as video and audio, various programs, and other data on a built-in hard disk, and reads them from the hard disk during playback.

The disk drive 945 records and reproduces signals with respect to the mounted optical disk. An optical disk such as a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.), a Blu-ray (registered trademark) disk, or the like.

The selector 946 selects one of the encoded bit streams from the tuner 941 or the encoder 943 and supplies it to either the HDD unit 944 or the disk drive 945 when recording video or audio. Further, the selector 946 supplies the encoded bit stream output from the HDD unit 944 or the disk drive 945 to the decoder 947 at the time of reproduction of video and audio.

The decoder 947 performs a decoding process on the encoded bit stream. The decoder 947 supplies the video data generated by performing the decoding process to the OSD unit 948. The decoder 947 outputs audio data generated by performing the decoding process.

The OSD unit 948 generates video data for displaying a menu screen for selecting an item and the like, and superimposes it on the video data output from the decoder 947 and outputs the video data.

A user interface unit 950 is connected to the control unit 949. The user interface unit 950 includes an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 949.

The control unit 949 is configured using a CPU, a memory, and the like. The memory stores programs executed by the CPU and various data necessary for the CPU to perform processing. The program stored in the memory is read and executed by the CPU at a predetermined timing such as when the recording / reproducing apparatus 940 is activated. The CPU executes the program to control each unit so that the recording / reproducing device 940 operates according to the user operation.

In the recording / reproducing apparatus configured as described above, the decoder 947 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, when the motion vector of the base layer image is used as the motion vector prediction vector candidate of the enhancement layer image, it is possible to decode the encoded stream that improves the encoding efficiency.

<Configuration example of imaging device>
FIG. 35 illustrates a schematic configuration of an imaging apparatus to which the present technology is applied. The imaging device 960 images a subject, displays an image of the subject on a display unit, and records it on a recording medium as image data.

The imaging device 960 includes an optical block 961, an imaging unit 962, a camera signal processing unit 963, an image data processing unit 964, a display unit 965, an external interface unit 966, a memory unit 967, a media drive 968, an OSD unit 969, and a control unit 970. Have. In addition, a user interface unit 971 is connected to the control unit 970. Furthermore, the image data processing unit 964, the external interface unit 966, the memory unit 967, the media drive 968, the OSD unit 969, the control unit 970, and the like are connected via a bus 972.

The optical block 961 is configured using a focus lens, a diaphragm mechanism, and the like. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 is configured using a CCD or CMOS image sensor, generates an electrical signal corresponding to the optical image by photoelectric conversion, and supplies the electrical signal to the camera signal processing unit 963.

The camera signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the electrical signal supplied from the imaging unit 962. The camera signal processing unit 963 supplies the image data after the camera signal processing to the image data processing unit 964.

The image data processing unit 964 performs an encoding process on the image data supplied from the camera signal processing unit 963. The image data processing unit 964 supplies the encoded data generated by performing the encoding process to the external interface unit 966 and the media drive 968. Further, the image data processing unit 964 performs a decoding process on the encoded data supplied from the external interface unit 966 and the media drive 968. The image data processing unit 964 supplies the image data generated by performing the decoding process to the display unit 965. Further, the image data processing unit 964 superimposes the processing for supplying the image data supplied from the camera signal processing unit 963 to the display unit 965 and the display data acquired from the OSD unit 969 on the image data. To supply.

The OSD unit 969 generates display data such as a menu screen and icons made up of symbols, characters, or figures and outputs them to the image data processing unit 964.

The external interface unit 966 includes, for example, a USB input / output terminal, and is connected to a printer when printing an image. In addition, a drive is connected to the external interface unit 966 as necessary, a removable medium such as a magnetic disk or an optical disk is appropriately mounted, and a computer program read from them is installed as necessary. Furthermore, the external interface unit 966 has a network interface connected to a predetermined network such as a LAN or the Internet. For example, the control unit 970 reads the encoded data from the media drive 968 in accordance with an instruction from the user interface unit 971, and supplies the encoded data to the other device connected via the network from the external interface unit 966. it can. Also, the control unit 970 may acquire encoded data and image data supplied from another device via the network via the external interface unit 966 and supply the acquired data to the image data processing unit 964. it can.

As the recording medium driven by the media drive 968, any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory is used. The recording medium may be any type of removable medium, and may be a tape device, a disk, or a memory card. Of course, a non-contact IC (Integrated Circuit) card may be used.

Further, the media drive 968 and the recording medium may be integrated and configured by a non-portable storage medium such as a built-in hard disk drive or an SSD (Solid State Drive).

The control unit 970 is configured using a CPU. The memory unit 967 stores a program executed by the control unit 970, various data necessary for the control unit 970 to perform processing, and the like. The program stored in the memory unit 967 is read out and executed by the control unit 970 at a predetermined timing such as when the imaging device 960 is activated. The control unit 970 controls each unit so that the imaging device 960 performs an operation according to a user operation by executing a program.

In the imaging apparatus configured as described above, the image data processing unit 964 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, when the motion vector of the base layer image is used as a candidate for the motion vector prediction vector of the enhancement layer image, the encoding efficiency can be improved. In addition, when the motion vector of the base layer image is used as a motion vector prediction vector candidate for the enhancement layer image, it is possible to decode an encoded stream that improves the encoding efficiency.

<Application example of scalable coding>
(First system)
Next, a specific usage example of scalable encoded data that has been encoded by the scalability function (hierarchical encoding) will be described. Scalable encoding is used for selection of data to be transmitted, for example, as in the example shown in FIG.

In the data transmission system 1000 shown in FIG. 36, the distribution server 1002 reads the scalable encoded data stored in the scalable encoded data storage unit 1001, and via the network 1003, the personal computer 1004, the AV device 1005, the tablet This is distributed to the terminal device such as the device 1006 and the mobile phone 1007.

At this time, the distribution server 1002 selects and transmits encoded data of appropriate quality according to the capability of the terminal device, the communication environment, and the like. Even if the distribution server 1002 transmits unnecessarily high-quality data, the terminal device does not always obtain a high-quality image, and may cause a delay or an overflow. Moreover, there is a possibility that the communication band is unnecessarily occupied or the load on the terminal device is unnecessarily increased. On the other hand, even if the distribution server 1002 transmits unnecessarily low quality data, there is a possibility that an image with sufficient image quality cannot be obtained in the terminal device. Therefore, the distribution server 1002 appropriately reads and transmits the scalable encoded data stored in the scalable encoded data storage unit 1001 as encoded data having an appropriate quality with respect to the capability and communication environment of the terminal device. .

For example, it is assumed that the scalable encoded data storage unit 1001 stores scalable encoded data (BL + EL) 1011 encoded in a scalable manner. The scalable encoded data (BL + EL) 1011 is encoded data including both a base layer and an enhancement layer, and is a data that can be decoded to obtain both a base layer image and an enhancement layer image. It is.

The distribution server 1002 selects an appropriate layer according to the capability of the terminal device that transmits data, the communication environment, and the like, and reads the data of the layer. For example, the distribution server 1002 reads high-quality scalable encoded data (BL + EL) 1011 from the scalable encoded data storage unit 1001 and transmits it to the personal computer 1004 and the tablet device 1006 with high processing capability as they are. . On the other hand, for example, the distribution server 1002 extracts base layer data from the scalable encoded data (BL + EL) 1011 for the AV device 1005 and the cellular phone 1007 having a low processing capability, and performs scalable encoding. Although it is data of the same content as the data (BL + EL) 1011, it is transmitted as scalable encoded data (BL) 1012 having a lower quality than the scalable encoded data (BL + EL) 1011.

By using scalable encoded data in this way, the amount of data can be easily adjusted, so that the occurrence of delay and overflow can be suppressed, and the unnecessary increase in the load on the terminal device and communication medium can be suppressed. be able to. In addition, since scalable encoded data (BL + EL) 1011 has reduced redundancy between layers, the amount of data can be reduced as compared with the case where encoded data of each layer is used as individual data. . Therefore, the storage area of the scalable encoded data storage unit 1001 can be used more efficiently.

Note that since various devices can be applied to the terminal device, such as the personal computer 1004 to the cellular phone 1007, the hardware performance of the terminal device varies depending on the device. Moreover, since the application which a terminal device performs is also various, the capability of the software is also various. Furthermore, the network 1003 serving as a communication medium can be applied to any communication network including wired, wireless, or both, such as the Internet and a LAN (Local Area Network), and has various data transmission capabilities. Furthermore, there is a risk of change due to other communications.

Therefore, the distribution server 1002 communicates with the terminal device that is the data transmission destination before starting data transmission, and the hardware performance of the terminal device, the performance of the application (software) executed by the terminal device, etc. Information regarding the capability of the terminal device and information regarding the communication environment such as the available bandwidth of the network 1003 may be obtained. The distribution server 1002 may select an appropriate layer based on the information obtained here.

Note that the layer extraction may be performed by the terminal device. For example, the personal computer 1004 may decode the transmitted scalable encoded data (BL + EL) 1011 and display a base layer image or an enhancement layer image. Further, for example, the personal computer 1004 extracts the base layer scalable encoded data (BL) 1012 from the transmitted scalable encoded data (BL + EL) 1011 and stores it or transfers it to another device. The base layer image may be displayed after decoding.

Of course, the numbers of the scalable encoded data storage unit 1001, the distribution server 1002, the network 1003, and the terminal devices are arbitrary. In the above, the example in which the distribution server 1002 transmits data to the terminal device has been described, but the usage example is not limited to this. The data transmission system 1000 may be any system as long as it transmits a scalable encoded data to a terminal device by selecting an appropriate layer according to the capability of the terminal device or a communication environment. Can be applied to the system.

(Second system)
Also, scalable coding is used for transmission via a plurality of communication media, for example, as in the example shown in FIG.

In the data transmission system 1100 shown in FIG. 37, a broadcasting station 1101 transmits base layer scalable encoded data (BL) 1121 by terrestrial broadcasting 1111. Also, the broadcast station 1101 transmits enhancement layer scalable encoded data (EL) 1122 via an arbitrary network 1112 including a wired or wireless communication network or both (for example, packetized transmission).

The terminal apparatus 1102 has a reception function of the terrestrial broadcast 1111 broadcast by the broadcast station 1101 and receives base layer scalable encoded data (BL) 1121 transmitted via the terrestrial broadcast 1111. The terminal apparatus 1102 further has a communication function for performing communication via the network 1112, and receives enhancement layer scalable encoded data (EL) 1122 transmitted via the network 1112.

The terminal device 1102 decodes the base layer scalable encoded data (BL) 1121 acquired via the terrestrial broadcast 1111 according to, for example, a user instruction, and obtains or stores a base layer image. Or transmit to other devices.

Also, the terminal device 1102, for example, in response to a user instruction, the base layer scalable encoded data (BL) 1121 acquired via the terrestrial broadcast 1111 and the enhancement layer scalable encoded acquired via the network 1112 Data (EL) 1122 is combined to obtain scalable encoded data (BL + EL), or decoded to obtain an enhancement layer image, stored, or transmitted to another device.

As described above, the scalable encoded data can be transmitted via a communication medium that is different for each layer, for example. Therefore, the load can be distributed, and the occurrence of delay and overflow can be suppressed.

Also, depending on the situation, the communication medium used for transmission may be selected for each layer. For example, scalable encoded data (BL) 1121 of a base layer having a relatively large amount of data is transmitted via a communication medium having a wide bandwidth, and scalable encoded data (EL) 1122 having a relatively small amount of data is transmitted. You may make it transmit via a communication medium with a narrow bandwidth. Further, for example, the communication medium for transmitting the enhancement layer scalable encoded data (EL) 1122 is switched between the network 1112 and the terrestrial broadcast 1111 according to the available bandwidth of the network 1112. May be. Of course, the same applies to data of an arbitrary layer.

By controlling in this way, an increase in load in data transmission can be further suppressed.

Of course, the number of layers is arbitrary, and the number of communication media used for transmission is also arbitrary. In addition, the number of terminal devices 1102 serving as data distribution destinations is also arbitrary. Furthermore, in the above description, broadcasting from the broadcasting station 1101 has been described as an example, but the usage example is not limited to this. The data transmission system 1100 can be applied to any system as long as it is a system that divides scalable encoded data into a plurality of layers and transmits them through a plurality of lines.

(Third system)
Further, scalable encoding is used for storing encoded data as in the example shown in FIG. 38, for example.

In the imaging system 1200 illustrated in FIG. 38, the imaging device 1201 performs scalable coding on image data obtained by imaging the subject 1211, and as scalable coded data (BL + EL) 1221, a scalable coded data storage device 1202. To supply.

The scalable encoded data storage device 1202 stores the scalable encoded data (BL + EL) 1221 supplied from the imaging device 1201 with quality according to the situation. For example, in the normal case, the scalable encoded data storage device 1202 extracts base layer data from the scalable encoded data (BL + EL) 1221, and the base layer scalable encoded data ( BL) 1222. On the other hand, for example, in the case of attention, the scalable encoded data storage device 1202 stores scalable encoded data (BL + EL) 1221 with high quality and a large amount of data.

By doing so, the scalable encoded data storage device 1202 can store an image with high image quality only when necessary, so that an increase in the amount of data can be achieved while suppressing a reduction in the value of the image due to image quality degradation. And the use efficiency of the storage area can be improved.

For example, assume that the imaging device 1201 is a surveillance camera. When the monitoring target (for example, an intruder) is not shown in the captured image (in the normal case), the content of the captured image is likely to be unimportant, so reduction of the data amount is given priority, and the image data (scalable coding) Data) is stored in low quality. On the other hand, when the monitoring target appears in the captured image as the subject 1211 (at the time of attention), since the content of the captured image is likely to be important, the image quality is given priority and the image data (scalable) (Encoded data) is stored with high quality.

Note that whether it is normal time or attention time may be determined by the scalable encoded data storage device 1202 analyzing an image, for example. Alternatively, the imaging apparatus 1201 may make a determination, and the determination result may be transmitted to the scalable encoded data storage device 1202.

It should be noted that the criterion for determining whether the time is normal or noting is arbitrary, and the content of the image as the criterion is arbitrary. Of course, conditions other than the contents of the image can also be used as the criterion. For example, it may be switched according to the volume or waveform of the recorded sound, may be switched at every predetermined time, or may be switched by an external instruction such as a user instruction.

In the above, an example of switching between the normal state and the attention state has been described. However, the number of states is arbitrary, for example, normal, slightly attention, attention, very attention, etc. Alternatively, three or more states may be switched. However, the upper limit number of states to be switched depends on the number of layers of scalable encoded data.

Also, the imaging apparatus 1201 may determine the number of layers for scalable coding according to the state. For example, in a normal case, the imaging apparatus 1201 may generate base layer scalable encoded data (BL) 1222 with low quality and a small amount of data, and supply the scalable encoded data storage apparatus 1202 to the scalable encoded data storage apparatus 1202. For example, when attention is paid, the imaging device 1201 generates scalable encoded data (BL + EL) 1221 having a high quality and a large amount of data, and supplies the scalable encoded data storage device 1202 to the scalable encoded data storage device 1202. May be.

In the above, the monitoring camera has been described as an example. However, the use of the imaging system 1200 is arbitrary and is not limited to the monitoring camera.

In the present specification, an example in which various types of information such as a base vector flag are multiplexed in an encoded stream and transmitted from the encoding side to the decoding side has been described. However, the method for transmitting such information is not limited to such an example. For example, these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream). Information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.

This technology receives bitstreams compressed by orthogonal transform such as discrete cosine transform and motion compensation, such as MPEG and H.26x, via network media such as satellite broadcasting, cable TV, the Internet, and mobile phones. The present invention can be applied to an encoding device or a decoding device that is used when processing on a storage medium such as an optical, magnetic disk, or flash memory.

In addition, in this specification, the case where encoding and decoding are performed using a method according to the HEVC method has been described as an example, but the scope of application of the present technology is not limited thereto. As long as the encoding target image is hierarchized and encoded using inter prediction, and a corresponding decoding device, it can be applied to encoding and decoding devices of other systems.

Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared by a plurality of devices via a network and is jointly processed.

Further, each step described in the above flowchart can be executed by one device or can be shared by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

Note that the present technology can be configured as follows.

(1)
A processing unit that performs compensation processing of an image of a first layer of an image having a hierarchical structure using a motion vector;
An image comprising: a list registration unit for registering motion vector prediction vector candidates of the first layer image including the motion vector of the second layer image in a list based on the characteristics of the image having the layer structure. Processing equipment.
(2)
The image processing apparatus according to (1), wherein the list registration unit registers the prediction vector candidates in the list based on a feature of the first layer image.
(3)
The image processing apparatus according to (1), wherein the list registration unit registers the prediction vector candidates in the list based on a feature of the second layer image.
(4)
The image processing apparatus according to (1), wherein the list registration unit registers the prediction vector candidates in the list based on characteristics of the images of the first and second hierarchies.
(5)
When the size of a PU (Prediction Unit) of an image having the hierarchical structure is equal to or larger than a predetermined size, the list registration unit determines a motion vector of the second layer image as a motion vector of the first layer image. The image processing apparatus according to any one of (1) to (4), wherein the image vector is registered in the list as a candidate for the prediction vector earlier.
(6)
When the size of a PU (Prediction Unit) of an image having the hierarchical structure is smaller than a predetermined size, the list registration unit determines a motion vector of the first layer image from a motion vector of the second layer image. The image processing apparatus according to any one of (1) to (5), wherein the image vector is registered in the list as a candidate for the prediction vector first.
(7)
The list registration unit registers the prediction vector candidates including the motion vector of the second layer image in the list when the size of a PU (Prediction Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size. The image processing apparatus according to any one of (1) to (4).
(8)
When the size of a CU (Coding Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size, the list registration unit determines the motion vector of the second layer image as the motion vector of the first layer image. The image processing device according to any one of (1) to (6), wherein the image vector is registered in the list as a candidate for the prediction vector earlier.
(9)
When the size of a CU (Coding Unit) of the image having the hierarchical structure is smaller than a predetermined size, the list registration unit uses the motion vector of the first layer image based on the motion vector of the second layer image. The image processing apparatus according to any one of (1) to (6) or (8), wherein the image is registered in the list as candidates for the prediction vector first.
(10)
The list registration unit registers the prediction vector candidates including the motion vector of the second layer image in the list when the size of a CU (Coding Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size. The image processing apparatus according to any one of (1) to (4) or (7).
(11)
When the resolution of the first layer image and the resolution of the second layer image are the same, the list registration unit obtains a motion vector of the second layer image as the first layer image. The image processing apparatus according to (1), wherein the motion vector is registered in the list as a candidate for the prediction vector before the motion vector.
(12)
When the resolution of the first layer image and the resolution of the second layer image are different from each other, the list registration unit determines the motion vector of the first layer image as the motion vector of the second layer image. The image processing apparatus according to (1) or (11), wherein the image processing apparatus is registered in the list as the prediction vector candidate earlier.
(13)
When the resolution of the image of the first layer and the resolution of the image of the second layer are the same, the list registration unit selects the prediction vector candidate including the motion vector of the image of the second layer. The image processing apparatus according to (1), wherein the image processing apparatus is registered in a list.
(14)
When the quantization parameter of the first layer image is larger than the quantization parameter of the second layer image, the list registration unit obtains a motion vector of the second layer image of the first layer. The image processing apparatus according to (1), wherein the image vector is registered in the list as a candidate for the prediction vector before an image motion vector.
(15)
When the quantization parameter of the second layer image is equal to or lower than the quantization parameter of the first layer image, the list registration unit obtains a motion vector of the first layer image from the second layer. The image processing apparatus according to (1) or (14), wherein the image vector is registered in the list as a candidate for the prediction vector prior to a motion vector of the image.
(16)
The list registration unit, when a quantization parameter of the first layer image is larger than a quantization parameter of the second layer image, the prediction vector candidate including a motion vector of the second layer image The image processing apparatus according to (1).
(17)
The list registration unit may include a first region or a second region that is temporally surrounding the block that is the target of the compensation processing in the first layer image based on the characteristics of the image having the hierarchical structure. The motion vector of one of the images is registered in the list as a candidate for the prediction vector, and the motion vector of the region of the second layer image corresponding to the other is registered in the list. The image processing apparatus according to any one of 16).
(18)
The image processing device according to any one of (1) to (17), further including: a setting unit configured to set identification information for identifying that the motion vector of the second layer image is a candidate for the prediction vector.
(19)
The image processing device
A processing step of performing compensation processing of an image of a first layer of an image having a hierarchical structure using a motion vector;
An image including a list registration step of registering, in a list, motion vector prediction vector candidates of the first layer image including a motion vector of the second layer image based on the characteristics of the image having the layer structure. Processing method.

10 encoding device, 21 setting unit, 131 processing unit, 133 list registration unit, 180 decoding device, 331 processing unit, 333 list registration unit

Claims

A processing unit that performs compensation processing of an image of a first layer of an image having a hierarchical structure using a motion vector;
An image comprising: a list registration unit for registering motion vector prediction vector candidates of the first layer image including the motion vector of the second layer image in a list based on the characteristics of the image having the layer structure. Processing equipment.
The image processing apparatus according to claim 1, wherein the list registration unit registers the prediction vector candidates in the list based on a feature of the image of the first hierarchy.
The image processing apparatus according to claim 1, wherein the list registration unit registers the prediction vector candidates in the list based on characteristics of the second layer image.
The image processing apparatus according to claim 1, wherein the list registration unit registers the prediction vector candidates in the list based on characteristics of the images of the first and second hierarchies.
When the size of a PU (Prediction Unit) of an image having the hierarchical structure is equal to or larger than a predetermined size, the list registration unit determines a motion vector of the second layer image as a motion vector of the first layer image. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in the list as the prediction vector candidate earlier.
When the size of a PU (Prediction Unit) of an image having the hierarchical structure is smaller than a predetermined size, the list registration unit determines a motion vector of the first layer image from a motion vector of the second layer image. The image processing apparatus according to claim 1, wherein the image processing apparatus is first registered in the list as a candidate for the prediction vector.
The list registration unit registers the prediction vector candidates including the motion vector of the second layer image in a list when the size of a PU (Prediction Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size. The image processing apparatus according to claim 1.
When the size of a CU (Coding Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size, the list registration unit determines the motion vector of the second layer image as the motion vector of the first layer image. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in the list as the prediction vector candidate earlier.
When the size of a CU (Coding Unit) of the image having the hierarchical structure is smaller than a predetermined size, the list registration unit uses the motion vector of the first layer image based on the motion vector of the second layer image. The image processing apparatus according to claim 1, wherein the image processing apparatus is first registered in the list as a candidate for the prediction vector.
The list registration unit registers the prediction vector candidates including the motion vector of the second layer image in the list when the size of a CU (Coding Unit) of the image having the hierarchical structure is equal to or larger than a predetermined size. The image processing apparatus according to claim 1.
When the resolution of the first layer image and the resolution of the second layer image are the same, the list registration unit obtains a motion vector of the second layer image as the first layer image. The image processing apparatus according to claim 1, wherein the image processing device is registered in the list as a candidate for the prediction vector prior to a motion vector.
When the resolution of the first layer image and the resolution of the second layer image are different from each other, the list registration unit determines the motion vector of the first layer image as the motion vector of the second layer image. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in the list as the prediction vector candidate earlier.
When the resolution of the image of the first layer and the resolution of the image of the second layer are the same, the list registration unit selects the prediction vector candidate including the motion vector of the image of the second layer. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in a list.
When the quantization parameter of the first layer image is larger than the quantization parameter of the second layer image, the list registration unit obtains a motion vector of the second layer image of the first layer. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in the list as a candidate for the prediction vector prior to an image motion vector.
When the quantization parameter of the second layer image is equal to or lower than the quantization parameter of the first layer image, the list registration unit obtains a motion vector of the first layer image from the second layer. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered in the list as a candidate for the prediction vector before a motion vector of the image.
The list registration unit, when a quantization parameter of the first layer image is larger than a quantization parameter of the second layer image, the prediction vector candidate including a motion vector of the second layer image The image processing apparatus according to claim 1.
The list registration unit may include a first region or a second region that is temporally surrounding the block that is the target of the compensation processing in the first layer image based on the characteristics of the image having the hierarchical structure. The motion vector of one of the images is registered in the list as a candidate for the prediction vector, and the motion vector of the region of the second layer image corresponding to the other is registered in the list. Image processing device.
The image processing apparatus according to claim 1, further comprising: a setting unit configured to set identification information for identifying that the motion vector of the second layer image is a candidate for the prediction vector.
The image processing device
A processing step of performing compensation processing of an image of a first layer of an image having a hierarchical structure using a motion vector;
An image including a list registration step of registering, in a list, motion vector prediction vector candidates of the first layer image including a motion vector of the second layer image based on the characteristics of the image having the layer structure. Processing method.