JP2019146225A - Image processing device and method - Google Patents

Abstract

To provide an image processing device and method which can suppress increase in processing time.SOLUTION: An image encoding device comprises: an encoding control part that controls whether a merge mode, in which an area to be processed is merged with a peripheral area positioned at the periphery of the area to be processed, is adopted or not, with respect to movement information, on the basis of information of the peripheral area belonging to the slice which the area to be processed belongs, in encoding which is performed independently for each slice in which a picture is divided into a plurality of pieces; and an encoding part that performs encoding of the area to be processed in the merge mode or in a mode other than the merge mode, according to control by the encoding control part.SELECTED DRAWING: Figure 13

Description

  The present disclosure relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method capable of suppressing an increase in processing time.

  In recent years, image information is handled as digital data, and MPEG (compressed by orthogonal transform such as discrete cosine transform and motion compensation is used for the purpose of efficient transmission and storage of information. A device conforming to a system such as Moving Picture Experts Group) is becoming popular in both information distribution at broadcasting stations and information reception in general households.

  In particular, MPEG2 (ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) 13818-2) is defined as a general-purpose image coding system, which includes both interlaced scanning images and progressive scanning images, as well as standard resolution images and This standard covers high-definition images and is currently widely used in a wide range of professional and consumer applications. By using the MPEG2 compression method, for example, a standard resolution interlaced scanning image having 720 × 480 pixels is 4 to 8 Mbps, and a high resolution interlaced scanning image having 1920 × 1088 pixels is 18 to 22 Mbps. (Bit rate) can be assigned to achieve a high compression rate and good image quality.

  MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but did not support encoding methods with a lower code amount (bit rate) than MPEG1, that is, a higher compression rate. With the widespread use of mobile terminals, the need for such an encoding system is expected to increase in the future, and the MPEG4 encoding system has been standardized accordingly. Regarding the image coding system, the standard was approved as an international standard as ISO / IEC 14496-2 in December 1998.

  Furthermore, in recent years, the standardization of the standard called H.26L (ITU-T (International Telecommunication Union Telecommunication Standardization Sector) Q6 / 16 VCEG (Video Coding Expert Group)) has progressed for the purpose of image coding for the initial video conference. Yes. H.26L is known to achieve higher encoding efficiency than the conventional encoding schemes such as MPEG2 and MPEG4, although a large amount of calculation is required for encoding and decoding. Currently, as part of MPEG4 activities, standardization to achieve higher coding efficiency based on this H.26L and incorporating functions not supported by H.26L is performed as Joint Model of Enhanced-Compression Video Coding. It has been broken.

  The standardization schedule became an international standard in March 2003 under the names of H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as AVC).

  However, the macroblock size of 16 pixels x 16 pixels is optimal for large image frames such as UHD (Ultra High Definition; 4000 pixels x 2000 pixels), which are subject to the next-generation encoding method. There was no fear.

  Therefore, for the purpose of further improving the encoding efficiency compared to AVC, HEVC (High Efficiency Video Coding) is being developed by JCTVC (Joint Collaboration Team-Video Coding), a joint standardization organization of ITU-T and ISO / IEC. Is being standardized (for example, see Non-Patent Document 1).

  In this HEVC encoding system, a coding unit (CU (Coding Unit)) is defined as a processing unit similar to a macroblock in AVC. The CU is not fixed to a size of 16 × 16 pixels like the AVC macroblock, and is specified in the image compression information in each sequence.

  By the way, in order to improve motion vector coding using median prediction in AVC, in addition to “Spatial Predictor” required by median prediction defined in AVC, “Temporal Predictor” and “Spatio-Temporal Predictor” It has been proposed to use any of them adaptively as predicted motion vector information (see, for example, Non-Patent Document 2).

  In the image information encoding apparatus, a cost function when each predicted motion vector information is used is calculated for each block, and optimal predicted motion vector information is selected. In the image compression information, flag information indicating information regarding which predicted motion vector information is used is transmitted to each block.

  In addition, a technique called Motion Partition Merging (hereinafter also referred to as merge mode) has been proposed as one of motion information encoding methods (see, for example, Non-Patent Document 3). In this method, when the motion information of the block is the same as the motion information of the neighboring blocks, only the flag information is transmitted, and when decoding, the motion information of the block is used using the motion information of the neighboring blocks. Is rebuilt.

  By the way, in the above-described image coding methods such as AVC and HEVC, for example, a method is provided in which a picture is divided into a plurality of slices and the processing is performed for each slice in order to parallelize the processing. In addition to such slices, entropy slices have also been proposed. An entropy slice is a processing unit for entropy encoding processing and entropy decoding processing. That is, in the entropy encoding process and the entropy decoding process, a picture is divided into a plurality of entropy slices and processed for each entropy slice. In the prediction process, this slice division is not applied for each picture. It is processed.

"Test Model under Consideration", JCTVC-B205, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG112nd Meeting: Geneva, CH, 21-28 July, 2010 Joel Jung, Guillaume Laroche, "Competition-Based Scheme for Motion Vector Selection and Coding", VCEG-AC06, ITU-Telecommunications Standardization SectorSTUDY GROUP 16 Question 6Video Coding Experts Group (VCEG) 29th Meeting: Klagenfurt, Austria, 17-18 July, 2006 Martin Winken, Sebastian Bosse, Benjamin Bross, Philipp Helle, Tobias Hinz, Heiner Kirchhoffer, Haricharan Lakshman, Detlev Marpe, Simon Oudin, Matthias Preiss, Heiko Schwarz, Mischa Siekmann, Karsten Suehring, and Thomas Wiegand, “Description of video coding technology proposed by Fraunhofer HHI ”, JCTVC-A116, April, 2010

  However, as described above, in the merge mode, it is necessary to refer to the motion information of the neighboring blocks when processing the motion information of the block that is the processing target. Therefore, when a picture is divided into a plurality of slices (including entropy slices) and processing is performed for each slice, a block of another slice may need to be referred to depending on the position of the block.

  In that case, since the processing of the block cannot be performed until the processing of the peripheral blocks is completed, it becomes impossible to parallelize the processing for each slice, and the throughput may be significantly reduced.

  The present disclosure has been made in view of such a situation, and processing time even when a merge mode is applied in encoding an image in which a picture is divided into a plurality of slices and processed in parallel for each slice. It is an object of the present invention to be able to suppress the increase of.

  One aspect of the present disclosure is that the number of motion vectors of a prediction unit that is a merge candidate while the peripheral prediction unit that does not belong to the processing target slice to which the processing target prediction unit belongs is not included in the merge candidate. An encoding control unit that controls encoding of merge mode based on the information, and an encoding unit that encodes the processing target prediction unit according to control of the encoding control unit. An image processing apparatus is provided.

  One aspect of the present disclosure is also an image processing method of an image processing device, and for motion information, a peripheral prediction unit that does not belong to a processing target slice to which a processing target prediction unit belongs is not included in a merge candidate. An image that sets information indicating the number of motion vectors of a prediction unit that is a candidate for merging, performs encoding control of merge mode based on the information, and encodes the processing target prediction unit according to the control It is a processing method.

  Another aspect of the present disclosure indicates the number of motion vectors included in a prediction unit that is a merge candidate while not including, as a merge candidate, a neighboring prediction unit that is adjacent to a processing target prediction unit at a slice boundary. An image comprising: an encoding control unit that sets information and controls encoding in merge mode based on the information; and an encoding unit that encodes the processing target prediction unit according to the control of the encoding control unit It is a processing device.

  Another aspect of the present disclosure is also an image processing method of an image processing apparatus, in which motion information is merged while a neighboring prediction unit adjacent to a processing target prediction unit at a slice boundary is not included in a merge candidate. An image processing method for setting information indicating the number of motion vectors of a candidate prediction unit, performing encoding control of merge mode based on the information, and encoding the processing target prediction unit according to the control It is.

  In one aspect of the present disclosure, with respect to motion information, while not including a peripheral prediction unit that does not belong to a processing target slice to which a processing target prediction unit belongs as a merge candidate, Information indicating the number is set, encoding control of the merge mode is performed based on the information, and the processing target prediction unit is encoded according to the control.

  In another aspect of the present disclosure, with respect to motion information, the number of motion vectors included in a merge candidate prediction unit is not included in a merge candidate while a neighboring prediction unit adjacent to a processing target prediction unit at a slice boundary is not included in the merge candidate. The information to be shown is set, the encoding control of the merge mode is performed based on the information, and the processing target prediction unit is encoded according to the control.

It is a block diagram which shows the main structural examples of an image coding apparatus. It is a figure explaining multi-slice. It is a figure explaining a coding unit. It is a figure explaining the relationship between a slice and a coding unit. It is a figure explaining the relationship between a slice and a coding unit. It is a figure explaining merge mode. It is a figure which shows the example of the position of the said area | region of a merge mode in multi slice, and a peripheral region. It is a figure which shows the other example of the position of the said area | region of a merge mode in multi slice, and a peripheral region. It is a figure which shows the further another example of the position of the said area | region of a merge mode in multi slice, and a periphery area | region. It is a figure which shows the further another example of the position of the said area | region of a merge mode in multi slice, and a periphery area | region. It is a figure which shows the example of the syntax of a coding unit. It is a figure which shows the example of the syntax of a prediction unit. It is a block diagram which shows the main structural examples of a lossless encoding part and an encoding control part. It is a block diagram which shows the main structural examples of an NMC setting part. It is a flowchart explaining the example of the flow of an encoding process. It is a flowchart explaining the example of the flow of a lossless encoding process. It is a flowchart explaining the example of the flow of a CU encoding process. It is a flowchart following FIG. 17 explaining the example of the flow of CU encoding processing. It is a flowchart explaining the example of the flow of a NumMergeCandidates setting process. It is a block diagram which shows the main structural examples of an image decoding apparatus. It is a block diagram which shows the main structural examples of a lossless decoding part and a decoding control part. It is a block diagram which shows the main structural examples of an NMC setting part. It is a flowchart explaining the example of the flow of a decoding process. It is a flowchart explaining the example of the flow of a lossless decoding process. It is a flowchart explaining the example of the flow of CU decoding process. It is a flowchart following FIG. 25 explaining the example of the flow of CU decoding processing. FIG. 26 is a block diagram illustrating a main configuration example of a personal computer. It is a block diagram which shows an example of a schematic structure of a television apparatus. It is a block diagram which shows an example of a schematic structure of a mobile telephone. It is a block diagram which shows an example of a schematic structure of a recording / reproducing apparatus. It is a block diagram which shows an example of a schematic structure of an imaging device.

Hereinafter, modes for carrying out the invention (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. First Embodiment (Image Encoding Device)
2. Second embodiment (image decoding apparatus)
3. Third embodiment (personal computer)
4). Fourth embodiment (television receiver)
5. Fifth embodiment (mobile phone)
6). Sixth embodiment (recording / reproducing apparatus)
7). Seventh embodiment (imaging device)

<1. First Embodiment>
[Image encoding device]
FIG. 1 is a block diagram illustrating a main configuration example of an image encoding device.

  The image encoding device 100 shown in FIG. Like the H.264 and MPEG (Moving Picture Experts Group) 4 Part 10 (AVC (Advanced Video Coding)) encoding schemes, the image data is encoded using prediction processing.

  As shown in FIG. 1, the image encoding device 100 includes an A / D conversion unit 101, a screen rearrangement buffer 102, a calculation unit 103, an orthogonal transformation unit 104, a quantization unit 105, a lossless encoding unit 106, and a storage buffer. 107. The image coding apparatus 100 also includes an inverse quantization unit 108, an inverse orthogonal transform unit 109, a calculation unit 110, a loop filter 111, a frame memory 112, a selection unit 113, an intra prediction unit 114, a motion prediction / compensation unit 115, and a prediction. An image selection unit 116 and a rate control unit 117 are included.

  The image encoding device 100 further includes an encoding control unit 121.

  The A / D conversion unit 101 performs A / D conversion on the input image data, and supplies the converted image data (digital data) to the screen rearrangement buffer 102 for storage. The screen rearrangement buffer 102 rearranges the images of the frames in the stored display order in the order of frames for encoding in accordance with the GOP (Group Of Picture), and rearranges the images in the order of the frames. This is supplied to the calculation unit 103. The screen rearrangement buffer 102 also supplies the image in which the order of the frames is rearranged to the intra prediction unit 114 and the motion prediction / compensation unit 115.

  The calculation unit 103 subtracts the prediction image supplied from the intra prediction unit 114 or the motion prediction / compensation unit 115 via the prediction image selection unit 116 from the image read from the screen rearrangement buffer 102, and the difference information Is output to the orthogonal transform unit 104.

  For example, in the case of an image on which inter coding is performed, the calculation unit 103 subtracts the predicted image supplied from the motion prediction / compensation unit 115 from the image read from the screen rearrangement buffer 102.

  The orthogonal transform unit 104 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 103. Note that this orthogonal transformation method is arbitrary. The orthogonal transform unit 104 supplies the transform coefficient to the quantization unit 105.

  The quantization unit 105 quantizes the transform coefficient supplied from the orthogonal transform unit 104. The quantization unit 105 sets a quantization parameter based on the information regarding the target value of the code amount supplied from the rate control unit 117, and performs the quantization. Note that this quantization method is arbitrary. The quantization unit 105 supplies the quantized transform coefficient to the lossless encoding unit 106.

  The lossless encoding unit 106 encodes the transform coefficient quantized by the quantization unit 105 using an arbitrary encoding method. Since the coefficient data is quantized under the control of the rate control unit 117, the code amount becomes a target value set by the rate control unit 117 (or approximates the target value).

  Further, the lossless encoding unit 106 acquires information indicating the mode of intra prediction from the intra prediction unit 114, and acquires information indicating the mode of inter prediction, motion vector information, and the like from the motion prediction / compensation unit 115. Further, the lossless encoding unit 106 acquires filter coefficients used in the loop filter 111 and the like.

  The lossless encoding unit 106 encodes these various types of information using an arbitrary encoding method, and sets (multiplexes) the information as part of the header information of the encoded data. The lossless encoding unit 106 supplies the encoded data obtained by encoding to the accumulation buffer 107 for accumulation.

  Examples of the encoding scheme of the lossless encoding unit 106 include variable length encoding or arithmetic encoding. Examples of variable length coding include H.264. CAVLC (Context-Adaptive Variable Length Coding) defined in the H.264 / AVC format. Examples of arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

  The accumulation buffer 107 temporarily holds the encoded data supplied from the lossless encoding unit 106. The accumulation buffer 107 outputs the stored encoded data to, for example, a recording device (recording medium) (not shown) or a transmission path (not shown) at a predetermined timing at a predetermined timing.

  The transform coefficient quantized by the quantization unit 105 is also supplied to the inverse quantization unit 108. The inverse quantization unit 108 inversely quantizes the quantized transform coefficient by a method corresponding to the quantization by the quantization unit 105. The inverse quantization method may be any method as long as it is a method corresponding to the quantization processing by the quantization unit 105. The inverse quantization unit 108 supplies the obtained transform coefficient to the inverse orthogonal transform unit 109.

  The inverse orthogonal transform unit 109 performs inverse orthogonal transform on the transform coefficient supplied from the inverse quantization unit 108 by a method corresponding to the orthogonal transform process by the orthogonal transform unit 104. The inverse orthogonal transform method may be any method as long as it corresponds to the orthogonal transform processing by the orthogonal transform unit 104. The inversely orthogonal transformed output (restored difference information) is supplied to the calculation unit 110.

  The calculation unit 110 is supplied from the intra prediction unit 114 or the motion prediction / compensation unit 115 via the prediction image selection unit 116 to the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 109, that is, the restored difference information. Predicted images are added to obtain a locally decoded image (decoded image). The decoded image is supplied to the loop filter 111 or the frame memory 112.

  The loop filter 111 includes a deblock filter, an adaptive loop filter, and the like, and appropriately performs a filtering process on the decoded image supplied from the calculation unit 110. For example, the loop filter 111 removes block distortion of the decoded image by performing a deblocking filter process on the decoded image. In addition, for example, the loop filter 111 performs image quality improvement by performing loop filter processing using a Wiener filter on the deblock filter processing result (decoded image from which block distortion has been removed). Do.

  Note that the loop filter 111 may perform arbitrary filter processing on the decoded image. Further, the loop filter 111 can supply information such as filter coefficients used for the filter processing to the lossless encoding unit 106 and encode it as necessary.

  The loop filter 111 supplies the filter process result (the decoded image after the filter process) to the frame memory 112. As described above, the decoded image output from the calculation unit 110 can be supplied to the frame memory 112 without passing through the loop filter 111. That is, the filter process by the loop filter 111 can be omitted.

  The frame memory 112 stores the supplied decoded image, and supplies the stored decoded image as a reference image to the selection unit 113 at a predetermined timing.

  The selection unit 113 selects a supply destination of the reference image supplied from the frame memory 112. For example, in the case of inter prediction, the selection unit 113 supplies the reference image supplied from the frame memory 112 to the motion prediction / compensation unit 115.

  The intra prediction unit 114 basically uses the pixel value in the processing target picture, which is a reference image supplied from the frame memory 112 via the selection unit 113, to generate a prediction image using a prediction unit (PU) as a processing unit. Perform intra prediction (intra-screen prediction) to be generated. The intra prediction unit 114 performs this intra prediction in a plurality of modes (intra prediction modes) prepared in advance.

  The intra prediction unit 114 generates predicted images in all candidate intra prediction modes, evaluates the cost function value of each predicted image using the input image supplied from the screen rearrangement buffer 102, and selects the optimum mode. select. When the intra prediction unit 114 selects the optimal intra prediction mode, the intra prediction unit 114 supplies the predicted image generated in the optimal mode to the predicted image selection unit 116.

  Further, as described above, the intra prediction unit 114 appropriately supplies the intra prediction mode information indicating the adopted intra prediction mode to the lossless encoding unit 106 and causes the encoding to be performed.

  The motion prediction / compensation unit 115 basically uses the input image supplied from the screen rearrangement buffer 102 and the reference image supplied from the frame memory 112 via the selection unit 113 as a processing unit. Motion prediction (inter prediction) is performed, motion compensation processing is performed according to the detected motion vector, and a predicted image (inter predicted image information) is generated. The motion prediction / compensation unit 115 performs such inter prediction in a plurality of modes (inter prediction modes) prepared in advance.

  The motion prediction / compensation unit 115 generates a prediction image in all candidate inter prediction modes, evaluates the cost function value of each prediction image, and selects an optimal mode. When the optimal inter prediction mode is selected, the motion prediction / compensation unit 115 supplies the predicted image generated in the optimal mode to the predicted image selection unit 116.

  In addition, the motion prediction / compensation unit 115 transmits information indicating the inter prediction mode employed, information necessary for performing processing in the inter prediction mode when decoding the encoded data, and the like. To be encoded.

  The predicted image selection unit 116 selects a supply source of the predicted image supplied to the calculation unit 103 or the calculation unit 110. For example, in the case of inter coding, the prediction image selection unit 116 selects the motion prediction / compensation unit 115 as a supply source of the prediction image, and calculates the prediction image supplied from the motion prediction / compensation unit 115 as the calculation unit 103 or the calculation unit. To the unit 110.

  The rate control unit 117 controls the quantization operation rate of the quantization unit 105 based on the code amount of the encoded data stored in the storage buffer 107 so that overflow or underflow does not occur.

  The encoding control unit 121 controls the encoding process of the lossless encoding unit 106. At that time, the encoding control unit 121 determines whether to perform encoding in the merge mode. In the determination, the encoding control unit 121 sets a parameter called NumMergeCandidates used for the determination. NumMergeCandidates is a parameter related to a motion vector included in a peripheral area that is located around the area to be processed and that may refer to a motion vector in the merge mode. When the parameter is set, the encoding control unit 121 determines whether or not the peripheral region is included in the same slice (the relevant slice) as the relevant region with respect to the multi-sliced processing target picture (the relevant picture). Confirm.

  That is, in the control of the encoding process of the lossless encoding unit 106, the encoding control unit 121 can use a peripheral area that can be referred to in the merge mode, or can not use it. Whether or not to enter the merge mode is determined in consideration of the existence of the merge mode, and the merge mode is controlled based on the determination result. The encoding control unit 121 also controls encoding processing for modes other than the merge mode, such as a skip mode, an intra prediction mode, an inter prediction mode, and a direct mode.

  The lossless encoding unit 106 performs a lossless encoding process in the mode selected by the encoding control unit 121.

[Multi-slice]
In an image encoding method such as MPEG2 or AVC, one picture can be divided into a plurality of slices, and each slice can be processed in parallel (multi-slice).

  In the case of MPEG2, as shown in the example of FIG. 2A, the maximum size of a slice is one macroblock line, and all slices constituting a B picture must be B slices.

  On the other hand, in the case of AVC, as shown in the example of FIG. 3B, the slice may be larger than one macroblock line, and the boundary of the slice may not be the right end (right end of the screen) of the macroblock line. A single picture may be composed of different types of slices.

  In the case of AVC, the deblocking filter process can be executed across slice boundaries. However, processing using adjacent information such as intra prediction, CABAC, CAVLC, and motion vector prediction cannot be executed across slice boundaries.

  In other words, since the encoding process of each slice can be performed independently of each other, it is possible to divide one picture into a plurality of slices and encode each slice in parallel. That is, such slice division can realize reduction in encoding processing time (enhancement of encoding processing).

[Coding unit]
By the way, in the AVC encoding method, a macro block and a sub macro block obtained by dividing the macro block into a plurality of units are used as processing units such as a prediction process and an encoding process. However, the macroblock size of 16 pixels x 16 pixels is optimal for large image frames such as UHD (Ultra High Definition; 4000 pixels x 2000 pixels), which are subject to the next-generation encoding method. is not.

  Therefore, for the purpose of further improving coding efficiency than AVC, it is a joint standardization organization of ITU-T (International Telecommunication Union Telecommunication Standardization Sector) and ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission). Standardization of a coding method called HEVC (High Efficiency Video Coding) is being promoted by a certain JCTVC (Joint Collaboration Team-Video Coding).

  In AVC, a hierarchical structure including macroblocks and sub-macroblocks is defined, but in HEVC, a coding unit (CU (Coding Unit)) is defined as shown in FIG.

  A CU is also called a Coding Tree Block (CTB), and is a partial area of a picture unit image that plays the same role as a macroblock in AVC. The latter is fixed to a size of 16 × 16 pixels, whereas the size of the former is not fixed, and is specified in the image compression information in each sequence.

  For example, in the sequence parameter set (SPS (Sequence Parameter Set)) included in the output encoded data, the maximum size (LCU (Largest Coding Unit)) and minimum size ((SCU (Smallest Coding Unit)) of the CU are specified. Is done.

  Within each LCU, split-flag = 1 can be divided into smaller CUs within a range that does not fall below the SCU size. In the example of FIG. 3, the LCU size is 128 and the maximum hierarchical depth is 5. When the value of split_flag is “1”, the 2N × 2N size CU is divided into N × N size CUs that are one level below.

  Furthermore, CU is divided into prediction units (Prediction Units (PU)) that are regions (partial regions of images in units of pictures) that are processing units for intra or inter prediction, and are regions that are processing units for orthogonal transformation It is divided into transform units (Transform Units (TU)), which are (partial regions of images in picture units). Currently, in HEVC, it is possible to use 16 × 16 and 32 × 32 orthogonal transforms in addition to 4 × 4 and 8 × 8.

  In the case of an encoding method in which a CU is defined and various processes are performed in units of the CU as in the above HEVC, it can be considered that a macroblock in AVC corresponds to an LCU. However, since the CU has a hierarchical structure as shown in FIG. 3, the size of the LCU in the highest hierarchy is generally set larger than the AVC macroblock, for example, 128 × 128 pixels. is there.

  The present disclosure can be applied to an encoding method using such a CU, PU, TU, and the like instead of a macroblock. That is, the processing unit for performing the prediction process may be an arbitrary area. That is, in the following description, only such macroblocks and sub-macroblocks are included in a region to be processed in prediction processing (also referred to as the region or a region of interest) or a peripheral region located around the region. CU, PU, TU, etc. are included.

  The LCUs (CU, PU, and TU) as described above are obtained by dividing the slice area into a plurality of parts and belong to the lower layer of the slice. That is, in the case of the multi-slice as described in FIG. 2, as shown in FIG. 4, the LCU is included in any slice.

  As shown in FIG. 5, the start address of the LCU is specified by a relative position from the start of each slice. Identification information and size are designated for each area (CU, PU, and TU) in the LCU. That is, the position of each area (for example, the start address) can be specified from the information. Therefore, the position of the area and the peripheral area, and the range of the slice can be easily specified from the information. In other words, whether or not the peripheral region belongs to the slice (whether it is available or unavailable) can be easily specified.

  Note that the slice boundaries can be set in units of PUs. That is, there may be an LCU positioned so as to straddle a plurality of slices. Also in this case, a region corresponding to each motion vector such as PU (region of prediction processing unit) is included in any one slice.

[Merge motion partition]
By the way, as one of the motion information encoding methods, a method called “Motion Partition Merging” (merge mode) as shown in FIG. 6 has been proposed. In this method, two flags, Merge_Flag and Merge_Left_Flag, are transmitted as merge information that is information related to the merge mode.

  Merge_Flag = 1 indicates that the motion information of the region X is the same as the motion information of the peripheral region T adjacent on the region or the peripheral region L adjacent to the left of the region. At this time, Merge_Left_Flag is included in the merge information and transmitted. Merge_Flag = 0 indicates that the motion information of the region X is different from the motion information of the peripheral region T and the peripheral region L. In this case, the motion information of the area X is transmitted.

  When the motion information of the region X is the same as the motion information of the peripheral region L, Merge_Flag = 1 and Merge_Left_Flag = 1. When the motion information of the region X is the same as the motion information of the peripheral region T, Merge_Flag = 1 and Merge_Left_Flag = 0.

[Multi-slice merge mode]
As described above, in the merge mode, the motion information of the peripheral area is referred to. However, in the case of the multi-slice as described above, there is a possibility that the peripheral region L and the peripheral region T that may be referred to in the merge mode are located in a different slice from the region X.

  FIG. 7 to FIG. 10 show examples of the positional relationship between the multi-slice, the area in the merge mode, and the peripheral area.

  FIG. 7 shows that the region (CU_X), the peripheral region (PU_T) adjacent to the region (CU_X), and the peripheral region (PU_L) adjacent to the left of the region (CU_X) are all one. A state of being located in (belonging to) the slice (slice 1) is shown.

  In this case, both the peripheral area (PU_T) and the peripheral area (PU_L) can be referred to (available).

  FIG. 8 shows that the region (CU_X) and the peripheral region (PU_L) are located in slice 1 (belong to the slice), but the peripheral region (PU_T) is located in slice 0 (does not belong to the slice) It shows a state.

  In this case, the peripheral area (PU_L) can be referred to (available), but the peripheral area (PU_T) cannot be referred to (unavailable).

  FIG. 9 shows that the region (CU_X) is located in slice 1 (belongs to the slice), but the peripheral region (PU_T) and the peripheral region (PU_L) are located in slice 0 (does not belong to the slice) It shows a state.

  In this case, both the peripheral area (PU_T) and the peripheral area (PU_L) cannot be referred to (unavailable).

  FIG. 10 shows that the region (CU_X) and the peripheral region (PU_T) are located in the slice 1 (belonging to the slice), but the peripheral region (PU_L) is located in the slice 0 (does not belong to the slice) It shows a state.

  In this case, the peripheral area (PU_T) can be referred to (available), but the peripheral area (PU_L) cannot be referred to (unavailable).

  Note that the slice boundary includes a picture edge in addition to the boundary between slices. What is important is whether or not the peripheral area is available, that is, whether or not the peripheral area is included in the slice. Therefore, the state where the peripheral area is unavailable includes not only the case where the peripheral area belongs to another slice but also the case where the peripheral area does not exist (will be located outside the picture). It is.

  In the example of FIG. 6, when the peripheral region L and the peripheral region T are unavailable as described above, the processing of the region X cannot proceed until the processing of the peripheral region L and the peripheral region T is completed. Therefore, in multi-slice, it is conceivable that processing is performed in parallel for each slice. However, parallel processing becomes difficult and processing time may increase due to the reference of the peripheral region.

[Syntax]
Further, not only when referring to the motion information of the peripheral area in the merge mode but also when determining whether or not to adopt the merge mode, the motion information of the peripheral area is required.

  FIG. 11 shows an example of CU syntax. FIG. 12 shows an example of PU syntax. In FIG. 11 and FIG. 12, the number at the left end of each row is the row number given for convenience of explanation.

  For example, as shown in the seventh line of FIG. 11 and the twelfth line of FIG. 12, the value of a parameter called NumMergeCandidates is used to determine whether or not to adopt the merge mode. This parameter indicates the count value (total number) of motion vectors included in a peripheral area (a candidate area to be merged with the area) that may be merged with the area in the merge mode.

  The encoding control unit 121 in FIG. 1 assumes that this NumMergeCandidates is greater than 0 as one of the conditions for adopting the merge mode. A value of 0 for NumMergeCandidates indicates that there is no region having motion information among the candidate regions to be merged with the region. In this case, since merging is impossible, the encoding control unit 121 controls to adopt a mode other than the merge mode.

  In other words, when NumMergeCandidates is greater than 0, there is at least one region having motion information among the candidate regions to be merged with the region, so the encoding control unit 121 determines other conditions related to the merge mode. I do.

  In order to accurately obtain the NumMergeCandidates, it is necessary to check the motion information of all the peripheral areas that are candidates for the area to be merged with the area. In other words, this NumMergeCandidates is necessary in determining whether or not to adopt the merge mode, and in order to obtain the value, it is necessary to refer to the motion information of the surrounding area. Therefore, as described above, when there is an unusable peripheral area, it is difficult to perform parallel processing in determining whether to adopt the merge mode, which may cause a delay.

[Calculation of NumMergeCandidates in the encoding controller]
Therefore, the encoding control unit 121 determines whether or not there is a motion vector only for the peripheral region existing in the slice, and obtains NumMergeCandidates. That is, the encoding control unit 121 counts the peripheral area (increments NumMergeCandidates) only when the peripheral area exists in the slice and has a motion vector.

  In this way, first, it is not necessary to refer to the motion vector of the peripheral area that does not belong to the slice in determining whether to adopt the merge mode. In addition, a peripheral region that does not belong to the slice is excluded from candidate regions to be merged with the region. In other words, only a peripheral region belonging to the slice is a candidate for a region to be merged with the region. Therefore, even when the merge mode is adopted, only the peripheral area belonging to the slice is merged into the area, so that it is not necessary to refer to the motion vector of the peripheral area that does not belong to the slice.

  Therefore, since the encoding control unit 121 and the lossless encoding unit 106 need only refer to the motion information in the slice, there is no need to wait until the processing of another slice is completed. Therefore, the image coding apparatus 100 can realize parallel processing for each slice, and can suppress an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode.

  Note that the encoding control unit 121 only has to calculate NumMergeCandidates as described above, and there is no need to change the syntax. Therefore, development is easy, and there is no fear of increasing the code amount or reducing versatility.

  The slice described above may be a processing unit that can divide a picture into a plurality of pieces and process them in parallel. Therefore, this slice includes, for example, an entropy slice in addition to a normal slice. Of course, the shape, number and position of slices are arbitrary. That is, the picture division position and the number of divisions are arbitrary.

  In the above description, slice 1 has been described as the slice, but the same applies to any slice that is the slice. For example, in the example of FIG. 4, the same applies to the case where the area in slice 0 or slice 2 is the area concerned.

  As described above, even when there is no peripheral area at the left end or the upper end of the picture, the encoding control unit 121 performs the motion vector of the peripheral area in the same manner as when the peripheral area exists in another slice. Do not increment NumMergeCandidates.

[Lossless encoding unit and encoding control unit]
FIG. 13 is a block diagram illustrating a main configuration example of the lossless encoding unit 106 and the encoding control unit 121.

  As shown in FIG. 13, the lossless encoding unit 106 includes a NAL (Network Abstraction Layer) encoding unit 131 and a CU data encoding unit 132.

  The NAL encoding unit 131 encodes NAL data such as a sequence parameter set (SPS (Sequence Parameter Set)), a picture parameter set (PPS (Picture Parameter Set)), and a slice header. The CU data encoding unit 132 encodes data of a hierarchy below the CU (VCL (Video Coding Layer)).

  The CU data encoding unit 132 includes a skip flag encoding unit 141, a skip mode encoding unit 142, a merge flag encoding unit 143, and a merge mode encoding unit 144. The CU data encoding unit 132 includes a PredMode encoding unit 145, an intra encoding unit 146, an inter encoding unit 147, and a direct mode encoding unit 148.

  The skip flag encoding unit 141 generates and encodes a skip flag indicating whether or not to adopt the skip mode in accordance with the control of the encoding control unit 121. The skip mode encoding unit 142 performs encoding processing in the skip mode according to the control of the encoding control unit 121.

  The merge flag encoding unit 143 generates and encodes a merge flag (MergeFlag) indicating whether or not to adopt the merge mode according to the control of the encoding control unit 121. The merge mode encoding unit 144 performs encoding processing in the merge mode according to the control of the encoding control unit 121.

  The PredMode encoding unit 145 encodes PredMode, which is a parameter indicating the prediction mode, under the control of the encoding control unit 121. The intra encoding unit 146 performs processing related to encoding of the difference image generated using the intra prediction according to the control of the encoding control unit 121. The inter coding unit 147 performs processing related to coding of a difference image generated using inter prediction according to the control of the coding control unit 121. The direct mode encoding unit 148 performs processing related to encoding of the difference image generated using the direct mode according to the control of the encoding control unit 121.

  As illustrated in FIG. 13, the encoding control unit 121 includes a slice determination unit 161, a skip flag determination unit 162, an NMC (NumMergeCandidates) setting unit 163, an NMC determination unit 164, a merge flag determination unit 165, and a PredMode determination. Part 166.

  The slice determination unit 161 determines the type of the slice, and supplies the determination result to the skip flag encoding unit 141 and the PredMode encoding unit 145 or the skip flag determination unit 162. The skip flag determination unit 162 determines the value (or presence) of the skip flag generated (or not generated) in the skip flag encoding unit 141, and the determination result is used as the skip mode encoding unit 142 or NMC. It supplies to the determination part 164.

  The NMC setting unit 163 obtains (sets) the value of the parameter NumMergeCandidates and supplies the value to the NMC determination unit 164. The NMC determination unit 164 determines the value of NumMergeCandidates and supplies the determination result to the merge flag encoding unit 143 or the merge flag determination unit 165.

  The merge flag determination unit 165 determines the value (or presence) of the merge flag generated (or not generated) in the merge flag encoding unit 143, and the determination result is the merge mode encoding unit 144 or slice determination unit. 161.

  The PredMode determining unit 166 determines the value (or presence) of the PredMode generated (or not) in the PredMode encoding unit 145, and determines the determination result as the intra encoding unit 146, the inter encoding unit 147, or This is supplied to the direct mode encoding unit 148.

[NMC setting part]
FIG. 14 is a block diagram illustrating a main configuration example of the NMC setting unit 163.

  As illustrated in FIG. 14, the NMC setting unit 163 includes an NMC reset unit 181, a position determination unit 182, a type determination unit 183, an NMC update unit 184, and an NMC holding unit 185.

  The NMC reset unit 181 resets the value of the parameter NumMergeCandidates held in the NMC holding unit 185 to 0.

  When the position determination unit 182 receives a reset notification from the NMC reset unit 181, receives a notification of updating the value of NumMergeCandidates from the NMC update unit 184, or acquires a determination result from the type determination unit 183, the position determination unit 182 Information on the position of the slice and the area X is acquired from the unit 131, the position of the area X is obtained, the positions of the peripheral area T and the peripheral area L are obtained from the position of the area X, and they exist in the slice It is determined whether or not. The position determination unit 182 supplies the determination result to the type determination unit 183.

  When the type determination unit 183 obtains the determination result from the position determination unit 182, the type determination unit 183 determines the prediction type of the peripheral region L and the peripheral region T existing in the slice from the CU data encoding unit 132. That is, the type determination unit 183 determines whether or not the surrounding area L and the surrounding area T have motion information. The type determination unit 183 supplies the determination result to the position determination unit 182 or the NMC update unit 184.

  Upon obtaining the determination result from the type determination unit 183, the NMC update unit 184 increments (+1) the value of the parameter NumMergeCandidates stored in the NMC storage unit 185. That is, the NMC update unit 184 increments (+1) the value of the parameter NumMergeCandidates held in the NMC holding unit 185 when the peripheral area located in the slice has motion information.

  The NMC holding unit 185 supplies the value of the held parameter NumMergeCandidates to the NMC determination unit 164 at a predetermined timing or in response to a request from the NMC determination unit 164.

[Flow of encoding process]
Next, the flow of each process executed by the image encoding device 100 as described above will be described. First, an example of the flow of encoding processing will be described with reference to the flowchart of FIG.

  In step S101, the A / D conversion unit 101 performs A / D conversion on the input image. In step S102, the screen rearrangement buffer 102 stores the A / D converted image, and rearranges the picture from the display order to the encoding order.

  In step S103, the intra prediction unit 114 performs intra prediction processing in the intra prediction mode. In step S104, the motion prediction / compensation unit 115 performs an inter motion prediction process for performing motion prediction and motion compensation in the inter prediction mode.

  In step S <b> 105, the predicted image selection unit 116 determines an optimal mode based on the cost function values output from the intra prediction unit 114 and the motion prediction / compensation unit 115. That is, the predicted image selection unit 116 selects one of the predicted image generated by the intra prediction unit 114 and the predicted image generated by the motion prediction / compensation unit 115.

  In step S106, the calculation unit 103 calculates a difference between the image rearranged by the process of step S102 and the predicted image selected by the process of step S105. The data amount of the difference data is reduced compared to the original image data. Therefore, the data amount can be compressed as compared with the case where the image is encoded as it is.

  In step S107, the orthogonal transform unit 104 performs orthogonal transform on the difference information generated by the process in step S106. Specifically, orthogonal transformation such as discrete cosine transformation and Karhunen-Loeve transformation is performed, and transformation coefficients are output.

  In step S108, the quantization unit 105 quantizes the orthogonal transform coefficient obtained by the process in step S107.

  The difference information quantized by the process of step S108 is locally decoded as follows. That is, in step S109, the inverse quantization unit 108 inversely quantizes the quantized orthogonal transform coefficient (also referred to as a quantization coefficient) generated by the process in step S108 with characteristics corresponding to the characteristics of the quantization unit 105. To do. In step S <b> 110, the inverse orthogonal transform unit 109 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by the process of step S <b> 107 with characteristics corresponding to the characteristics of the orthogonal transform unit 104.

  In step S111, the calculation unit 110 adds the predicted image to the locally decoded difference information, and generates a locally decoded image (an image corresponding to the input to the calculation unit 103). In step S112, the loop filter 111 appropriately performs a loop filter process including a deblock filter process and an adaptive loop filter process on the locally decoded image obtained by the process of step S111.

  In step S113, the frame memory 112 stores the decoded image that has been subjected to the loop filter process by the process of step S112. It should be noted that an image that has not been filtered by the loop filter 111 is also supplied from the calculation unit 110 and stored in the frame memory 112.

  In step S114, the lossless encoding unit 106 encodes the transform coefficient quantized by the process in step S108. That is, lossless encoding such as variable length encoding or arithmetic encoding is performed on the difference image.

  The lossless encoding unit 106 encodes the quantization parameter calculated in step S108 and adds it to the encoded data. Further, the lossless encoding unit 106 encodes information related to the prediction mode of the prediction image selected by the process of step S105, and adds the encoded information to the encoded data obtained by encoding the difference image. That is, the lossless encoding unit 106 also encodes and encodes the optimal intra prediction mode information supplied from the intra prediction unit 114 or information according to the optimal inter prediction mode supplied from the motion prediction / compensation unit 115, and the like. Append to data.

  In step S115, the accumulation buffer 107 accumulates the encoded data obtained by the process in step S114. The encoded data stored in the storage buffer 107 is appropriately read and transmitted to the decoding side via a transmission path or a recording medium.

  In step S116, the rate control unit 117 causes the quantization unit 105 to prevent overflow or underflow based on the code amount (generated code amount) of the encoded data accumulated in the accumulation buffer 107 by the process of step S115. Controls the rate of quantization operation.

  When the process of step S116 ends, the encoding process ends.

  In step S <b> 114, the lossless encoding unit 106 performs an encoding process according to the control of the encoding control unit 121.

[Flow of lossless encoding processing]
Next, an example of the flow of lossless encoding processing executed in step S114 in FIG. 15 will be described with reference to the flowchart in FIG. As shown in FIG. 16, the lossless encoding process is performed for each image hierarchy.

  That is, the NAL encoding unit 131 generates and encodes an SPS in step S121, generates and encodes a PPS in step S122, and generates and encodes a slice header in step S123. In step S124, the CU data encoding unit 132 encodes the CU to be processed.

  The CU data encoding unit 132 repeats the process of step S124 for all the CUs in the slice that is the processing target. If it is determined in step S125 that there is no unprocessed CU in the slice, the CU data encoding unit 132 proceeds with the process to step S126.

  The NAL encoding unit 131 repeats the processing from step S123 to step S125 for all slices in the current picture to be processed. If it is determined in step S126 that there is no unprocessed slice in the picture, the NAL encoding unit 131 advances the process to step S127.

  The NAL encoding unit 131 repeats the processing from step S122 to step S126 for all the pictures in the sequence to be processed. If it is determined in step S127 that there is no unprocessed picture in the sequence, the NAL encoding unit 131 ends the lossless encoding process and returns the process to FIG.

[Flow of CU encoding process]
Next, an example of the flow of the CU encoding process executed in step S124 in FIG. 16 will be described with reference to the flowcharts in FIGS.

  When the CU encoding process is started, the slice determination unit 161 determines the type of the slice from the NAL data generated by the NAL encoding unit 131 in step S131, and whether or not the slice is an I slice. Determine whether. Only when the slice is not an I slice (P slice or B slice), the skip flag encoding unit 141 generates and encodes a skip flag in step S132.

  When the skip flag determination unit 162 determines that the value of the skip flag is 1 in step S133, the skip mode encoding unit 142 that has acquired the determination result from the skip flag determination unit 162 determines that the skip mode in step S134. To encode the CU data. When encoding ends, the CU encoding process ends, and the process returns to FIG.

  In addition, when the skip flag determination unit 162 determines in step S133 in FIG. 17 that the value of the skip flag is 0 or the skip flag does not exist, the skip flag determination unit 162 advances the processing to step S135. . In this case, encoding in the skip mode is not performed.

  In step S135, the NMC setting unit 163 sets NumMergeCandidates.

  Only when the NMC determining unit 164 determines in step S136 that the value of NumMergeCandidates set in step S135 is greater than 0, the merge flag encoding unit 143 generates and encodes a merge flag in step S137. .

  When the merge flag determination unit 165 determines that the value of the merge flag is 1 in step S138, the merge mode encoding unit 144 that has acquired the determination result from the merge flag determination unit 165 performs the CU in merge mode in step S139. Encode the data. When encoding ends, the CU encoding process ends, and the process returns to FIG.

  Also, in step S138 of FIG. 17, if the merge flag determination unit 165 determines that the value of the merge flag is 0 or that the merge flag does not exist, the process proceeds to the flowchart of FIG. Encoding according to the mode is performed.

  That is, only when the slice determination unit 161 determines in step S141 in FIG. 18 that the slice to be processed is not an I slice, the PredMode encoding unit 145 indicates a parameter indicating the type of prediction mode of the slice in step S142. Is generated and encoded.

  In step S143, when the PredMode determination unit 166 refers to PredMode and determines that the prediction mode of the region is the intra prediction mode, the intra encoding unit 146 encodes the CU data in the intra prediction mode in step S144. Turn into. That is, difference image information (quantized orthogonal transform coefficient), information on the intra prediction mode, and the like are encoded. When encoding ends, the CU encoding process ends, and the process returns to FIG.

  In addition, when the PredMode determination unit 166 determines that the prediction mode of the region is not the intra prediction mode but the inter prediction mode (step S143 and step S145), the inter coding unit 147 performs the inter prediction in step S146. Encode mode CU data. That is, difference image information (quantized orthogonal transform coefficient), information on the inter prediction mode, and the like are encoded. When encoding ends, the CU encoding process ends, and the process returns to FIG.

  Furthermore, when the PredMode determining unit 166 determines that the prediction mode of the region is not the intra prediction mode and the inter prediction mode (step S143 and step S145), the direct mode encoding unit 148 performs direct processing in step S147. CU data in prediction mode is encoded. When encoding ends, the CU encoding process ends, and the process returns to FIG.

[Flow of NumMergeCandidates setting process]
Next, an example of the flow of NumMergeCandidates setting processing will be described with reference to the flowchart of FIG.

  When the NumMergeCandidates setting process is started, in step S151, the NMC reset unit 181 resets the parameter NumMergeCandidates held in the NMC holding unit 185 to the initial value “0”.

  Only when the position determination unit 182 determines in step S152 that the peripheral region PU_L exists in the slice, and in step S153, the type determination unit 183 determines that the prediction mode of the peripheral region PU_L is not intra prediction. In step S154, the NMC update unit 184 increments (+1) the value of the parameter NumMergeCandidates held in the NMC holding unit 185.

  On the other hand, in step S152, the position determination unit 182 determines that the peripheral region PU_L does not exist in the slice, or in step S153, the type determination unit 183 determines that the prediction mode of the peripheral region PU_L is intra prediction. Or the value of the parameter NumMergeCandidates is not incremented (+1).

  Similar processing is performed for the peripheral region PU_T (steps S155 to S157).

  In step S158, the NMC holding unit 185 supplies the value of the held NumMergeCandidates to the NMC determination unit 164. When the value of NumMergeCandidates is output, the NumMergeCandidates setting process is terminated, and the process returns to FIG.

  By performing various processes as described above, the image encoding apparatus 100 can realize parallel processing for each slice, and suppress an increase in processing time due to generation of unnecessary delay time in processing related to the merge mode. can do.

<2. Second Embodiment>
[Image decoding device]
FIG. 20 is a block diagram illustrating a main configuration example of an image decoding device. The image decoding apparatus 200 shown in FIG. 20 decodes the encoded data generated by the image encoding apparatus 100 by a decoding method corresponding to the encoding method. Note that, similarly to the image encoding device 100, the image decoding device 200 performs a prediction process for each arbitrary region (for example, a prediction unit (PU)).

  As shown in FIG. 20, the image decoding apparatus 200 includes a storage buffer 201, a lossless decoding unit 202, an inverse quantization unit 203, an inverse orthogonal transform unit 204, a calculation unit 205, a loop filter 206, a screen rearrangement buffer 207, and a D A / A converter 208 is included. The image decoding apparatus 200 includes a frame memory 209, a selection unit 210, an intra prediction unit 211, a motion prediction / compensation unit 212, and a selection unit 213.

  Furthermore, the image decoding device 200 includes a decoding control unit 221.

  The accumulation buffer 201 accumulates the transmitted encoded data and supplies the encoded data to the lossless decoding unit 202 at a predetermined timing. The lossless decoding unit 202 decodes the information supplied from the accumulation buffer 201 and encoded by the lossless encoding unit 106 in FIG. 1 by a method corresponding to the encoding method of the lossless encoding unit 106. The lossless decoding unit 202 supplies the quantized coefficient data of the difference image obtained by decoding to the inverse quantization unit 203.

  Further, the lossless decoding unit 202 determines whether the intra prediction mode is selected as the optimal prediction mode or the inter prediction mode, and uses the intra prediction unit 211 and the motion prediction / compensation unit as information on the optimal prediction mode. The data is supplied to the mode determined to be selected from among 212. That is, for example, when the inter prediction mode is selected as the optimal prediction mode in the image encoding device 100, information regarding the optimal prediction mode is supplied to the motion prediction / compensation unit 212.

  The inverse quantization unit 203 inversely quantizes the quantized coefficient data obtained by decoding by the lossless decoding unit 202 using a method corresponding to the quantization method of the quantization unit 105 in FIG. Data is supplied to the inverse orthogonal transform unit 204.

  The inverse orthogonal transform unit 204 performs inverse orthogonal transform on the coefficient data supplied from the inverse quantization unit 203 in a method corresponding to the orthogonal transform method of the orthogonal transform unit 104 in FIG. The inverse orthogonal transform unit 204 obtains decoded residual data corresponding to the residual data before being orthogonally transformed in the image coding apparatus 100 by the inverse orthogonal transform process.

  Decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 205. In addition, a prediction image is supplied to the calculation unit 205 from the intra prediction unit 211 or the motion prediction / compensation unit 212 via the selection unit 213.

  The computing unit 205 adds the decoded residual data and the predicted image, and obtains decoded image data corresponding to the image data before the predicted image is subtracted by the computing unit 103 of the image encoding device 100. The arithmetic unit 205 supplies the decoded image data to the loop filter 206.

  The loop filter 206 appropriately performs a loop filter process including a deblock filter process and an adaptive loop filter process on the supplied decoded image, and supplies it to the screen rearrangement buffer 207.

  The loop filter 206 includes a deblock filter, an adaptive loop filter, and the like, and appropriately performs a filtering process on the decoded image supplied from the arithmetic unit 205. For example, the loop filter 206 removes block distortion of the decoded image by performing a deblocking filter process on the decoded image. In addition, for example, the loop filter 206 performs image quality improvement by performing loop filter processing using the Wiener Filter on the deblock filter processing result (decoded image from which block distortion has been removed). Do.

  Note that the loop filter 206 may perform arbitrary filter processing on the decoded image. Further, the loop filter 206 may perform filter processing using the filter coefficient supplied from the image encoding device 100 of FIG.

  The loop filter 206 supplies the filter processing result (the decoded image after the filter processing) to the screen rearrangement buffer 207 and the frame memory 209. Note that the decoded image output from the calculation unit 205 can be supplied to the screen rearrangement buffer 207 and the frame memory 209 without going through the loop filter 206. That is, the filter process by the loop filter 206 can be omitted.

  The screen rearrangement buffer 207 rearranges images. That is, the order of frames rearranged for the encoding order by the screen rearrangement buffer 102 in FIG. 1 is rearranged in the original display order. The D / A conversion unit 208 D / A converts the image supplied from the screen rearrangement buffer 207, outputs it to a display (not shown), and displays it.

  The frame memory 209 stores the supplied decoded image, and the stored decoded image is referred to as a reference image at a predetermined timing or based on an external request such as the intra prediction unit 211 or the motion prediction / compensation unit 212. To the selection unit 210.

  The selection unit 210 selects a supply destination of the reference image supplied from the frame memory 209. The selection unit 210 supplies the reference image supplied from the frame memory 209 to the intra prediction unit 211 when decoding an intra-coded image. The selection unit 210 also supplies the reference image supplied from the frame memory 209 to the motion prediction / compensation unit 212 when decoding an inter-coded image.

  Information indicating the intra prediction mode obtained by decoding the header information is appropriately supplied from the lossless decoding unit 202 to the intra prediction unit 211. The intra prediction unit 211 performs intra prediction using the reference image acquired from the frame memory 209 in the intra prediction mode used in the intra prediction unit 114 in FIG. 1, and generates a predicted image. The intra prediction unit 211 supplies the generated predicted image to the selection unit 213.

  The motion prediction / compensation unit 212 obtains information obtained by decoding the header information (optimum prediction mode information, difference information, code number of prediction motion vector information, etc.) from the lossless decoding unit 202.

  The motion prediction / compensation unit 212 performs inter prediction using the reference image acquired from the frame memory 209 in the inter prediction mode used in the motion prediction / compensation unit 115 in FIG. 1, and generates a predicted image.

  The decoding control unit 221 controls the decoding process of the lossless decoding unit 202. The lossless decoding unit 202 basically performs decoding processing by a method corresponding to the lossless encoding unit 106 in FIG. 1, and therefore, the control method of the decoding control unit 221 is basically the encoding control unit 121 in FIG. 1. This is the same as the control method. By aligning the control methods on the encoding side and the decoding side, the decoding control unit 221 can select a decoding method corresponding to the encoding method selected by the encoding control unit 121 so that the decoding process is performed correctly. Can be controlled.

  That is, the decoding control unit 221 determines whether to perform decoding in the merge mode. At the time of the determination, the decoding control unit 221 sets a parameter called NumMergeCandidates. When the parameter is set, the decoding control unit 221 confirms whether or not a peripheral region is included in the slice for the multi-sliced picture.

  That is, the decoding control unit 221 determines whether the peripheral area that can be referred to in the merge mode is available (available) or unavailable (unavailable) in the decoding process control of the lossless decoding unit 202. It is determined whether or not the merge mode is set in consideration, and the merge mode is controlled based on the determination result. The decoding control unit 221 also controls the decoding process for modes other than the merge mode, such as a skip mode, an intra prediction mode, an inter prediction mode, and a direct mode.

  The lossless decoding unit 202 performs lossless decoding processing in the mode selected by the decoding control unit 221.

  By doing so, the decoding control unit 221 and the lossless decoding unit 202 need only refer to the motion information in the slice, and therefore do not need to wait until the processing of another slice is completed. Therefore, the image decoding apparatus 200 can realize parallel processing for each slice, and can suppress an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode.

[Reversible decoding unit and decoding control unit]
FIG. 21 is a block diagram illustrating a main configuration example of the lossless decoding unit 202 and the decoding control unit 221.

  As shown in FIG. 21, the lossless decoding unit 202 includes a NAL decoding unit 231 and a CU data decoding unit 232.

  The NAL decoding unit 231 decodes NAL encoded data such as a sequence parameter set, a picture parameter set, and a slice header. The CU data decoding unit 232 decodes the encoded data in the hierarchy below the CU.

  The CU data decoding unit 232 includes a skip flag decoding unit 241, a skip mode decoding unit 242, a merge flag decoding unit 243, and a merge mode decoding unit 244. The CU data decoding unit 232 includes a PredMode decoding unit 245, an intra decoding unit 246, an inter decoding unit 247, and a direct mode decoding unit 248.

  The skip flag decoding unit 241 decodes the skip flag according to the control of the decoding control unit 221. The skip mode decoding unit 242 performs a decoding process in the skip mode according to the control of the decoding control unit 221.

  The merge flag decoding unit 243 decodes the merge flag (MergeFlag) according to the control of the decoding control unit 221. The merge mode decoding unit 244 performs a decoding process in the merge mode according to the control of the decoding control unit 221.

  The PredMode decoding unit 245 decodes PredMode according to the control of the decoding control unit 221. The intra decoding unit 246 performs processing related to decoding of encoded data of the difference image generated using intra prediction according to the control of the decoding control unit 221. The inter decoding unit 247 performs processing related to decoding of encoded data of a difference image generated using inter prediction according to the control of the decoding control unit 221. The direct mode decoding unit 248 performs processing related to decoding of encoded data of the difference image generated using the direct mode according to the control of the decoding control unit 221.

  In addition, the decoding control unit 221 performs basically the same control as the encoding control unit 121. That is, as illustrated in FIG. 21, the decoding control unit 221 includes a slice determination unit 261, a skip flag determination unit 262, an NMC setting unit 263, an NMC determination unit 264, a merge flag determination unit 265, and a PredMode determination unit 266. .

  Each of the slice determination unit 261 through PredMode determination unit 266 performs basically the same processing as the slice determination unit 261 through PredMode determination unit 166 of the encoding control unit 121.

[NMC setting part]
FIG. 22 is a block diagram illustrating a main configuration example of the NMC setting unit 263.

  The NMC setting unit 263 performs basically the same processing as the NMC setting unit 163. That is, the NMC setting unit 263 includes an NMC reset unit 281, a position determination unit 282, a type determination unit 283, an NMC update unit 284, and an NMC holding unit 285, as shown in FIG.

  The NMC reset unit 281 to NMC holding unit 285 perform basically the same processing as the NMC reset unit 181 to NMC 185, respectively.

[Decoding process flow]
Next, the flow of each process executed by the image decoding apparatus 200 as described above will be described. First, an example of the flow of decoding processing will be described with reference to the flowchart of FIG.

  When the decoding process is started, in step S201, the accumulation buffer 201 accumulates the transmitted code stream. In step S202, the lossless decoding unit 202 decodes the code stream supplied from the accumulation buffer 201. That is, the I picture, P picture, and B picture encoded by the lossless encoding unit 106 in FIG. 1 are decoded. Various information other than the difference image information included in the code stream, such as difference motion information, code number of predicted motion vector information, and merge information, is also decoded.

  In step S203, the inverse quantization unit 203 inversely quantizes the quantized orthogonal transform coefficient obtained by the process in step S202. In step S204, the inverse orthogonal transform unit 204 performs inverse orthogonal transform on the orthogonal transform coefficient inversely quantized in step S203.

  In step S205, the intra prediction unit 211 or the motion prediction / compensation unit 212 performs a prediction process using the supplied information. In step S206, the selection unit 213 selects the predicted image generated in step S205. In step S207, the calculation unit 205 adds the predicted image selected in step S206 to the difference image information obtained by the inverse orthogonal transform in step S204. Thereby, a decoded image is obtained.

  In step S208, the loop filter 206 appropriately performs a loop filter process including a deblock filter process and an adaptive loop filter process on the decoded image obtained in step S207.

  In step S209, the screen rearrangement buffer 207 rearranges the images filtered in step S208. That is, the order of frames rearranged for encoding by the screen rearrangement buffer 102 of the image encoding device 100 is rearranged to the original display order.

  In step S210, the D / A conversion unit 208 performs D / A conversion on the image in which the frame order is rearranged in step S209. This image is output to a display (not shown), and the image is displayed.

  In step S211, the frame memory 209 stores the image filtered in step S208. In step S205, this image is used as a reference image for generating a predicted image.

  When the process of step S211 ends, the decoding process ends.

[Flow of lossless decoding]
Next, an example of the flow of lossless decoding processing executed in step S202 of FIG. 23 will be described with reference to the flowchart of FIG.

  This lossless decoding process is performed for each image layer, as in the case of the lossless encoding process.

  That is, the NAL decoding unit 231 decodes the SPS encoded data in step S221, decodes the PPS encoded data in step S222, and decodes the slice header encoded data in step S223. In step S224, the CU data decoding unit 232 decodes the CU to be processed.

  The CU data decoding unit 232 repeats the process of step S224 for all CUs in the slice that is the processing target. If it is determined in step S225 that there is no unprocessed CU in the slice, the CU data decoding unit 232 advances the process to step S226.

  The NAL decoding unit 231 repeats the processing from step S223 to step S225 for all slices in the picture to be processed. If it is determined in step S226 that there is no unprocessed slice in the picture, the NAL decoding unit 231 advances the process to step S227.

  The NAL decoding unit 231 repeats the processing from step S222 to step S226 for all the pictures in the sequence to be processed. If it is determined in step S227 that there is no unprocessed picture in the sequence, the NAL decoding unit 231 ends the lossless decoding process and returns the process to FIG.

[CU decoding process]
Next, an example of the flow of the CU decoding process executed in step S224 of FIG. 24 will be described with reference to the flowcharts of FIGS.

  When the CU decoding process is started, the slice determination unit 261 determines the type of the slice from the NAL data decoded by the NAL decoding unit 231 in step S231, and determines whether the slice is an I slice. judge.

  If the slice is not an I slice (P slice or B slice), the skip flag decoding unit 241 decodes the skip flag in step S232. If it is determined that the slice is an I slice, the skip flag is not encoded, so this process is omitted.

  If the skip flag determination unit 262 determines in step S233 that a skip flag exists and its value is 1, the skip mode decoding unit 242 decodes CU data in the skip mode in step S234. When the CU data is decoded, the skip mode decoding unit 242 ends the CU decoding process and returns the process to FIG.

  In step S233, if the skip flag determination unit 262 determines that the skip flag does not exist or the value thereof is 0, the NMC setting unit 263 sets NumMergeCandidates in step S235. Since this NumMergeCandidates setting process is performed in the same manner as described with reference to the flowchart of FIG. 19, the detailed description of the NumMergeCandidates setting process is omitted.

  When NumMergeCandidates is set and the NMC determination unit 264 determines in step S236 that the value of the NumMergeCandidates is greater than 0, the merge flag decoding unit 243 decodes the merge flag in step S237. If it is determined that the value of NumMergeCandidates is 0, the merge flag is not encoded, so this process is omitted.

  In step S238, when the merge flag determination unit 265 determines that the merge flag exists and its value is 1, the merge mode decoding unit 244 decodes the CU data in the merge mode in step S239. When the CU data is decoded, the merge mode decoding unit 244 ends the CU decoding process and returns the process to FIG.

  In step S238, if the merge flag determination unit 265 determines that the merge flag does not exist or the value thereof is 0, the process proceeds to FIG.

  In this case, the CU data is decoded by a method according to the prediction mode. That is, if the slice determination unit 261 determines in step S241 in FIG. 26 that the slice is not an I slice, the PredMode decoding unit 245 decodes pred_mode in step S242. If it is determined that the slice is an I slice, pred_mode is not encoded, and thus this process is omitted.

  In step S243, when the PredMode determination unit 266 determines that the prediction mode of the region is the intra prediction mode, the intra decoding unit 246 decodes in the intra prediction mode (encoded in the intra prediction mode) in step S244. CU data is decrypted in an appropriate manner). When the CU data is decoded, the intra decoding unit 246 ends the CU decoding process and returns the process to FIG.

  Also, when the PredMode determination unit 266 determines that the prediction mode of the region is not the intra prediction mode in step S243 and determines that the prediction mode is the inter prediction mode in step S245, the inter decoding unit 247 determines that the inter decoding unit 247 in step S246 Decode in prediction mode (decode CU data encoded in inter prediction mode with an appropriate method). When the CU data is decoded, the inter decoding unit 247 ends the CU decoding process and returns the process to FIG.

  Furthermore, when the PredMode determination unit 266 determines that the prediction mode of the region is not the intra prediction mode in step S243 and determines that the prediction mode is not the inter prediction mode in step S245, the direct mode decoding unit 248 determines in step S247 that Decode in direct prediction mode (decode CU data encoded in direct prediction mode with an appropriate method). When the CU data is decoded, the direct mode decoding unit 248 ends the CU decoding process and returns the process to FIG.

  As described above, by performing various processes, the decoding control unit 221 and the lossless decoding unit 202 need only refer to the motion information in the slice, and thus need to wait until the processing of other slices is completed. Absent. Therefore, the image decoding apparatus 200 can realize parallel processing for each slice, and can suppress an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode.

  Note that the present technology is, for example, MPEG, H.264, and the like. When receiving image information (bitstream) compressed by orthogonal transformation such as discrete cosine transformation and motion compensation, such as 26x, via network media such as satellite broadcasting, cable television, the Internet, or mobile phones. The present invention can be applied to an image encoding device and an image decoding device used in the above. In addition, the present technology can be applied to an image encoding device and an image decoding device that are used when processing on a storage medium such as an optical, magnetic disk, and flash memory. Furthermore, the present technology can also be applied to motion prediction / compensation devices included in such image encoding devices and image decoding devices.

<3. Third Embodiment>
[Personal computer]
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

  In FIG. 27, a CPU (Central Processing Unit) 501 of the personal computer 500 performs various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 513 to a RAM (Random Access Memory) 503. Execute the process. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output interface 510 is also connected to the bus 504.

  The input / output interface 510 includes an input unit 511 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), an output unit 512 including a speaker, and a hard disk. A communication unit 514 including a storage unit 513 and a modem is connected. The communication unit 514 performs communication processing via a network including the Internet.

  A drive 515 is connected to the input / output interface 510 as necessary, and a removable medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 513 as necessary.

  When the above-described series of processing is executed by software, a program constituting the software is installed from a network or a recording medium.

  For example, as shown in FIG. 27, the recording medium is distributed to distribute the program to the user separately from the apparatus main body, and includes a magnetic disk (including a flexible disk) on which the program is recorded, an optical disk ( It only consists of removable media 521 consisting of CD-ROM (compact disc-read only memory), DVD (including digital versatile disc), magneto-optical disc (including MD (mini disc)), or semiconductor memory. Rather, it is composed of a ROM 502 on which a program is recorded and a hard disk included in the storage unit 513, which is distributed to the user in a state of being pre-installed in the apparatus main body.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  Further, in this specification, the system represents the entire apparatus composed of a plurality of devices (apparatuses).

  In addition, in the above description, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit). Of course, a configuration other than that described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). . That is, the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present technology.

  An image encoding device and an image decoding device according to the above-described embodiments include a transmitter or a receiver in optical broadcasting, satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication, etc. The present invention can be applied to various electronic devices such as a recording device that records an image on a medium such as a magnetic disk and a flash memory, or a playback device that reproduces an image from these storage media. Hereinafter, four application examples will be described.

<4. Fourth Embodiment>
[First application example: television receiver]
FIG. 28 shows an example of a schematic configuration of a television apparatus to which the above-described embodiment is applied. The television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, And a bus 912.

  The tuner 902 extracts a signal of a desired channel from a broadcast signal received via the antenna 901, and demodulates the extracted signal. Then, the tuner 902 outputs the encoded bit stream obtained by the demodulation to the demultiplexer 903. In other words, the tuner 902 serves as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

  The demultiplexer 903 separates the video stream and audio stream of the viewing target program from the encoded bit stream, and outputs each separated stream to the decoder 904. Also, the demultiplexer 903 extracts auxiliary data such as EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control unit 910. Note that the demultiplexer 903 may perform descrambling when the encoded bit stream is scrambled.

  The decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. Then, the decoder 904 outputs the video data generated by the decoding process to the video signal processing unit 905. In addition, the decoder 904 outputs audio data generated by the decoding process to the audio signal processing unit 907.

  The video signal processing unit 905 reproduces the video data input from the decoder 904 and causes the display unit 906 to display the video. In addition, the video signal processing unit 905 may cause the display unit 906 to display an application screen supplied via a network. Further, the video signal processing unit 905 may perform additional processing such as noise removal on the video data according to the setting. Further, the video signal processing unit 905 may generate a GUI (Graphical User Interface) image such as a menu, a button, or a cursor, and superimpose the generated image on the output image.

  The display unit 906 is driven by a drive signal supplied from the video signal processing unit 905, and displays a video on a video screen of a display device (for example, a liquid crystal display, a plasma display, or an OELD (Organic ElectroLuminescence Display) (organic EL display)). Or an image is displayed.

  The audio signal processing unit 907 performs reproduction processing such as D / A conversion and amplification on the audio data input from the decoder 904 and outputs audio from the speaker 908. The audio signal processing unit 907 may perform additional processing such as noise removal on the audio data.

  The external interface 909 is an interface for connecting the television device 900 to an external device or a network. For example, a video stream or an audio stream received via the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also has a role as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

  The control unit 910 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, EPG data, data acquired via a network, and the like. For example, the program stored in the memory is read and executed by the CPU when the television apparatus 900 is activated. The CPU executes the program to control the operation of the television device 900 according to an operation signal input from the user interface 911, for example.

  The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches for the user to operate the television device 900, a remote control signal receiving unit, and the like. The user interface 911 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 910.

  The bus 912 connects the tuner 902, the demultiplexer 903, the decoder 904, the video signal processing unit 905, the audio signal processing unit 907, the external interface 909, and the control unit 910 to each other.

  In the television apparatus 900 configured as described above, the decoder 904 has the function of the image decoding apparatus according to the above-described embodiment. Thereby, when decoding an image in the television apparatus 900, parallel processing for each slice can be realized, and an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode can be suppressed. .

<5. Fifth embodiment>
[Second application example: mobile phone]
FIG. 29 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied. A cellular phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, a control unit 931, an operation A portion 932 and a bus 933.

  The antenna 921 is connected to the communication unit 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 933 connects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the demultiplexing unit 928, the recording / reproducing unit 929, the display unit 930, and the control unit 931 to each other.

  The mobile phone 920 has various operation modes including a voice call mode, a data communication mode, a shooting mode, and a videophone mode, and is used for sending and receiving voice signals, sending and receiving e-mail or image data, taking images, and recording data. Perform the action.

  In the voice call mode, an analog voice signal generated by the microphone 925 is supplied to the voice codec 923. The audio codec 923 converts an analog audio signal into audio data, A / D converts the compressed audio data, and compresses it. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 decompresses the audio data and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  Further, in the data communication mode, for example, the control unit 931 generates character data that constitutes an e-mail in response to an operation by the user via the operation unit 932. In addition, the control unit 931 causes the display unit 930 to display characters. In addition, the control unit 931 generates e-mail data in response to a transmission instruction from the user via the operation unit 932, and outputs the generated e-mail data to the communication unit 922. The communication unit 922 encodes and modulates email data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to restore the email data, and outputs the restored email data to the control unit 931. The control unit 931 displays the content of the electronic mail on the display unit 930 and stores the electronic mail data in the storage medium of the recording / reproducing unit 929.

  The recording / reproducing unit 929 has an arbitrary readable / writable storage medium. For example, the storage medium may be a built-in storage medium such as a RAM or a flash memory, and is externally mounted such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB (Unallocated Space Bitmap) memory, or a memory card. It may be a storage medium.

  In the shooting mode, for example, the camera unit 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the storage / playback unit 929.

  Further, in the videophone mode, for example, the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and the multiplexed stream is the communication unit 922. Output to. The communication unit 922 encodes and modulates the stream and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. These transmission signal and reception signal may include an encoded bit stream. Then, the communication unit 922 demodulates and decodes the received signal to restore the stream, and outputs the restored stream to the demultiplexing unit 928. The demultiplexing unit 928 separates the video stream and the audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, and a series of images is displayed on the display unit 930. The audio codec 923 decompresses the audio stream and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  In the mobile phone 920 configured as described above, the image processing unit 927 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Thereby, parallel processing of each slice can be realized when encoding and decoding an image on the mobile phone 920, and an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode can be suppressed. Can do.

<6. Sixth Embodiment>
[Third application example: recording / reproducing apparatus]
FIG. 30 shows an example of a schematic configuration of a recording / reproducing apparatus to which the above-described embodiment is applied. For example, the recording / reproducing device 940 encodes audio data and video data of a received broadcast program and records the encoded data on a recording medium. In addition, the recording / reproducing device 940 may encode audio data and video data acquired from another device and record them on a recording medium, for example. In addition, the recording / reproducing device 940 reproduces data recorded on the recording medium on a monitor and a speaker, for example, in accordance with a user instruction. At this time, the recording / reproducing device 940 decodes the audio data and the video data.

  The recording / reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface. 950.

  The tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Then, the tuner 941 outputs the encoded bit stream obtained by the demodulation to the selector 946. That is, the tuner 941 has a role as a transmission unit in the recording / reproducing apparatus 940.

  The external interface 942 is an interface for connecting the recording / reproducing apparatus 940 to an external device or a network. The external interface 942 may be, for example, an IEEE1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission unit in the recording / reproducing device 940.

  The encoder 943 encodes video data and audio data when the video data and audio data input from the external interface 942 are not encoded. Then, the encoder 943 outputs the encoded bit stream to the selector 946.

  The HDD 944 records an encoded bit stream in which content data such as video and audio are compressed, various programs, and other data on an internal hard disk. Further, the HDD 944 reads out these data from the hard disk when reproducing video and audio.

  The disk drive 945 performs recording and reading of data with respect to the mounted recording medium. The recording medium mounted on the disk drive 945 is, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.) or a Blu-ray (registered trademark) disk. It may be.

  The selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video and audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 during video and audio reproduction.

  The decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. The decoder 904 outputs the generated audio data to an external speaker.

  The OSD 948 reproduces the video data input from the decoder 947 and displays the video. Further, the OSD 948 may superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.

  The control unit 949 includes a processor such as a CPU, and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the recording / reproducing apparatus 940 is activated, for example. The CPU controls the operation of the recording / reproducing device 940 according to an operation signal input from the user interface 950, for example, by executing the program.

  The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for the user to operate the recording / reproducing device 940, a remote control signal receiving unit, and the like. The user interface 950 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.

  In the recording / reproducing apparatus 940 configured as described above, the encoder 943 has the function of the image encoding apparatus according to the above-described embodiment. The decoder 947 has the function of the image decoding apparatus according to the above-described embodiment. Thereby, when encoding and decoding an image in the recording / reproducing apparatus 940, parallel processing for each slice can be realized, and an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode is suppressed. be able to.

<7. Seventh Embodiment>
[Fourth Application Example: Imaging Device]
FIG. 31 illustrates an example of a schematic configuration of an imaging apparatus to which the above-described embodiment is applied. The imaging device 960 images a subject to generate an image, encodes the image data, and records it on a recording medium.

  The imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a bus. 972.

  The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 972 connects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970 to each other.

  The optical block 961 includes a focus lens and a diaphragm mechanism. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 includes an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and converts an optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. Then, the imaging unit 962 outputs the image signal to the signal processing unit 963.

  The signal processing unit 963 performs various camera signal processes such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data after the camera signal processing to the image processing unit 964.

  The image processing unit 964 encodes the image data input from the signal processing unit 963, and generates encoded data. Then, the image processing unit 964 outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes encoded data input from the external interface 966 or the media drive 968 to generate image data. Then, the image processing unit 964 outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may display the image by outputting the image data input from the signal processing unit 963 to the display unit 965. Further, the image processing unit 964 may superimpose display data acquired from the OSD 969 on an image output to the display unit 965.

  The OSD 969 generates a GUI image such as a menu, a button, or a cursor, for example, and outputs the generated image to the image processing unit 964.

  The external interface 966 is configured as a USB input / output terminal, for example. The external interface 966 connects the imaging device 960 and a printer, for example, when printing an image. Further, a drive is connected to the external interface 966 as necessary. For example, a removable medium such as a magnetic disk or an optical disk is attached to the drive, and a program read from the removable medium can be installed in the imaging device 960. Further, the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has a role as a transmission unit in the imaging device 960.

  The recording medium mounted on the media drive 968 may be any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. Further, a recording medium may be fixedly attached to the media drive 968, and a non-portable storage unit such as a built-in hard disk drive or an SSD (Solid State Drive) may be configured.

  The control unit 970 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the imaging device 960 is activated, for example. For example, the CPU controls the operation of the imaging device 960 according to an operation signal input from the user interface 971 by executing the program.

  The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960. The user interface 971 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 970.

  In the imaging device 960 configured as described above, the image processing unit 964 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Thereby, when encoding and decoding an image in the imaging device 960, parallel processing for each slice can be realized, and an increase in processing time due to generation of an unnecessary delay time in processing related to the merge mode can be suppressed. Can do.

  In the present specification, an example has been described in which various pieces of information such as prediction mode information and merge information are multiplexed on the header of the encoded stream and transmitted from the encoding side to the decoding side. However, the method for transmitting such information is not limited to such an example. For example, these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream). Information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present disclosure belongs can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present disclosure.

In addition, this technique can also take the following structures.
(1) In encoding performed independently for each slice that divides a picture into a plurality of images, a merge mode is adopted in which the region to be processed is merged with a peripheral region located around the region for motion information. An encoding control unit that controls whether or not based on information of a peripheral region belonging to the slice to which the region belongs,
An image processing apparatus comprising: an encoding unit that performs encoding of the region in the merge mode or other modes according to the control of the encoding control unit.
(2) The image processing device according to (1), wherein the encoding control unit adopts the merge mode when at least one of the peripheral regions belonging to the slice has motion information.
(3) The encoding control unit
A calculation unit that calculates the number of motion information included in the peripheral region belonging to the slice;
A determination unit that determines whether or not the number of pieces of motion information included in the peripheral area calculated by the calculation unit is greater than 0;
The image processing apparatus according to (1) or (2), further including: a control unit that adopts the merge mode when the determination unit determines that the number of motion information included in the peripheral area is greater than zero.
(4) The calculation unit includes:
A position determination unit that determines whether each peripheral area belongs to the slice;
A type determination unit that determines the type of prediction of the surrounding area determined to belong to the slice by the position determination unit;
An update unit that updates a value of a parameter that counts the number of pieces of motion information included in the peripheral region when the type determination unit determines the type of prediction of the peripheral region and determines that the type includes motion information. The image processing apparatus according to 3).
(5) The image processing apparatus according to any one of (1) to (4), further including a prediction processing unit that performs prediction processing for generating a predicted image independently for each slice.
(6) The image processing device according to any one of (1) to (4), wherein the slice is an entropy slice that divides only the encoding process performed on the picture by the encoding unit into a plurality of sections.
(7) An image processing method for an image processing apparatus,
In encoding performed independently for each slice that divides a picture into a plurality of slices, the encoding mode is a merge mode in which the area to be processed is merged with a peripheral area located around the area for motion information. Is controlled based on the information of the peripheral region belonging to the slice to which the region belongs,
An image processing method in which the encoding unit encodes the area in the merge mode or other modes according to the control.
(8) In decoding performed independently for each slice that divides a picture into multiple pieces, whether or not to adopt a merge mode for merging the region to be processed with a peripheral region located around the region for motion information A decoding control unit that controls whether or not based on information of a peripheral region belonging to the slice to which the region belongs,
An image processing apparatus comprising: a decoding unit that performs encoding of the region in a merge mode or other modes under the control of the decoding control unit.
(9) The image processing device according to (8), wherein the decoding control unit adopts the merge mode when at least one of the peripheral regions belonging to the slice has motion information.
(10) The decoding control unit
A calculation unit that calculates the number of motion information included in the peripheral region belonging to the slice;
A determination unit that determines whether or not the number of pieces of motion information included in the peripheral area calculated by the calculation unit is greater than 0;
The image processing apparatus according to (8) or (9), further including: a control unit that adopts the merge mode when the determination unit determines that the number of motion information included in the peripheral area is greater than zero.
(11) The calculation unit includes:
A position determination unit that determines whether each peripheral area belongs to the slice;
A type determination unit that determines the type of prediction of the surrounding area determined to belong to the slice by the position determination unit;
An update unit that updates a value of a parameter that counts the number of pieces of motion information included in the peripheral region when the type determination unit determines the type of prediction of the peripheral region and determines that the type includes motion information. The image processing apparatus according to 10).
(12) The image processing device according to any one of (8) to (11), further including a prediction processing unit that performs prediction processing for generating a predicted image independently for each slice.
(13) The image processing device according to any one of (8) to (11), wherein the slice is an entropy slice that divides only the decoding processing on the picture by the decoding unit into a plurality of pieces.
(14) An image processing method for an image processing apparatus,
In decoding performed independently for each slice that divides a picture into a plurality of slices, the decoding control unit adopts a merge mode for merging the region to be processed with a peripheral region located around the region for motion information Whether or not to control based on the information of the peripheral region belonging to the slice to which the region belongs,
An image processing method in which the decoding unit performs decoding of the area in the merge mode or other modes according to the control.

  100 image encoding device, 106 lossless encoding unit, 121 encoding control unit, 143 merge flag encoding unit, 144 merge mode encoding unit, 163 NMC setting unit, 164 NMC determination unit, 165 merge flag determination unit, 181 NMC Reset unit, 182 position determination unit, 183 type determination unit, 184 NMC update unit, 185 NMC holding unit, 200 image decoding device, 202 lossless decoding unit, 221 decoding control unit, 243 merge flag decoding unit, 244 merge mode decoding unit, 263 NMC setting unit, 264 NMC determination unit, 265 merge flag determination unit, 281 NMC reset unit, 282 position determination unit, 283 type determination unit, 284 NMC update unit, 285 NMC holding unit

Claims (9)

For motion information, while not including peripheral prediction units that do not belong to the processing target slice to which the processing target prediction unit belongs as merge candidates, information indicating the number of motion vectors that the merge candidate prediction unit has, An encoding control unit that controls encoding in the merge mode based on the information;
An image processing apparatus comprising: an encoding unit that encodes the processing target prediction unit according to control of the encoding control unit. A determination unit that determines whether the peripheral prediction unit belongs to the processing target slice to which the processing target prediction unit belongs;
The image processing apparatus according to claim 1, wherein the encoding control unit performs control so that the peripheral prediction unit determined not to belong to the processing target slice by the determination unit is not included in the merge candidates. The image processing apparatus according to claim 1, wherein the information is information used in a syntax of the processing target prediction unit. The image processing apparatus according to claim 1, wherein the encoding unit encodes the processing target prediction unit in a merge mode or other modes according to control of the encoding control unit. The image processing apparatus according to claim 1, further comprising: a prediction processing unit that performs prediction processing for generating a predicted image independently for each slice. The image processing apparatus according to claim 1, wherein the slice is an entropy slice that divides only a coding process on a picture by the coding unit into a plurality of pieces. For motion information, while not including peripheral prediction units that do not belong to the processing target slice to which the processing target prediction unit belongs as merge candidates, information indicating the number of motion vectors that the merge candidate prediction unit has, Based on the information, control the encoding of the merge mode,
An image processing method for encoding the processing target prediction unit according to the control. For the motion information, setting information indicating the number of motion vectors of the prediction unit of the merge candidate while not including the neighboring prediction unit adjacent to the processing target prediction unit at the slice boundary in the merge candidate, An encoding control unit for controlling encoding of the merge mode based on;
An image processing apparatus comprising: an encoding unit that encodes the processing target prediction unit according to control of the encoding control unit. For the motion information, setting information indicating the number of motion vectors of the prediction unit of the merge candidate while not including the neighboring prediction unit adjacent to the processing target prediction unit at the slice boundary in the merge candidate, Based on the merge mode encoding control,
An image processing method for encoding the processing target prediction unit according to the control.

Images