JP6419297B2 - Image encoding device, image encoding method and program, image decoding device, image decoding method and program - Google Patents

Description

  The present invention relates to encoding and decoding of layers having different spatial resolution and image quality. In particular, the present invention relates to an image encoding and decoding technique for dividing an image into a plurality of regions and encoding and decoding each divided region in a moving image.

  As an encoding method for compression recording of moving images, H.264 is used. H.264 / MPEG-4 AVC (hereinafter H.264) is known.

  In recent years, H.C. As a successor to H.264, an international standardization of a more efficient coding system was started, and JCT-VC (Joint Collaborative Team on Video Coding) was established between ISO / IEC and ITU-T. In this JCT-VC, standardization of the High Efficiency Video Coding encoding method (hereinafter referred to as HEVC) is underway (Non-Patent Document 1).

  HEVC employs a technique called a tile division method in which an image is divided into rectangular regions (tiles), and each region is independently encoded and decoded. Furthermore, in the tile division method, using Motion Constrained Tile Sets (hereinafter referred to as MCTS) composed of one or more tiles, it has been proposed to independently encode and decode the MCTS without depending on other tiles. (Non-Patent Document 2). In the proposal described in Non-Patent Document 2, it is defined that the MCTS can be set in sequence units. That is, in the same sequence, the MCTS positions in each frame are relatively equal. And in said proposal, when encoding and decoding MCTS in the frame of a process target, the pixel group of the position relatively equal to the said MCTS in another frame is made into the object of inter-frame prediction. That is, pixels other than the pixel group are not used as reference pixels to be referenced in the motion vector search. Thereby, the independence of encoding and decoding in MCTS can be ensured. Note that the position of the tile included in the MCTS in the image is encoded by being included in a SEI (Supplemental Enhancement Information) message.

  On the other hand, in the standardization of HEVC, expansion to hierarchical coding is also being studied. In hierarchical encoding, tiles to be encoded are encoded in the base layer and the enhancement layer, respectively. Then, the bit stream is generated by multiplexing the tiles encoded in each layer. In the hierarchical encoding as described above, the boundary position of the base layer tile and the boundary position of the enhancement layer tile can be set independently. When encoding a tile to be encoded in the enhancement layer, it is necessary to refer to the corresponding tile in the base layer, and thus it is necessary to specify the position of the tile in the base layer. Therefore, it has been proposed to use a tile_boundaries_aligned_flag code as a VUI (Video Usability Information) parameter (vui_parameters) of the enhancement layer (Non-patent Document 3). This code is obtained by encoding coincidence information that indicates whether or not the relative positions of tiles coincide between the layers. When the code is 1, it is guaranteed that the position of the tile boundary of the enhancement layer matches the position of the corresponding tile boundary of the base layer. As a result, the position of the base layer tile called in the encoding and decoding of the enhancement layer tile can be specified, so that the enhancement layer tile can be encoded and decoded independently, enabling high-speed encoding and decoding. To. Note that the base layer is the highest layer, and the subsequent extension layers are sequentially lower layers.

ITU-TH. 265 (04/2013) High efficiency video coding JCT-VC Contribution JCTVC-M0235 Internet <http: // phenix. int-evry. fr / jct / doc_end_user / documents / 13_Incheon / wg11 /> JCT-VC Contribution JCTVC-M0202 Internet <http: // phenix. int-evry. fr / jct / doc_end_user / documents / 13_Incheon / wg11 />

  However, the MCTS described in Non-Patent Document 2 does not consider hierarchical coding. That is, when the tile boundary and the MCTS position can be set for each layer, the relative positions of the tiles in each layer may be inconsistent. For example, when a predetermined tile of the enhancement layer is included in the MCTS and a tile at a position corresponding to the predetermined tile of the basic layer is not included in the MCTS, the basic layer corresponds to the predetermined tile. The surrounding tiles other than the position also need to be decoded.

  Here, it demonstrates concretely using FIG. FIG. 13 shows a state of tile division. Reference numerals 1301 to 1310 denote frames. Each of the frames 1301 to 1310 includes 12 tiles having tile numbers 0 to 11, respectively. Hereinafter, the tile with tile number 1 is referred to as tile 1. The same is true even if the number changes. For the sake of explanation, in the base layer, the tile division of each frame is divided into two in the horizontal direction and not divided in the vertical direction. Furthermore, in the enhancement layer, the tile division of each frame is assumed to be divided into four in the horizontal direction and three in the vertical direction. A thin frame in the figure represents a tile boundary.

  Each frame 1301, 1303, 1305, 1307 represents a frame of each layer at time t. A frame 1301 represents a frame of the base layer at time t. A frame 1305 represents a frame in the extended first layer (first extended layer) at time t. A frame 1303 represents a frame obtained by enlarging the reconstructed image obtained by locally decoding the frame 1301 to the resolution of the first enhancement layer. A frame 1309 represents a frame in the extended second layer (second extended layer) at time t. A frame 1307 represents a frame obtained by enlarging the decoded image of the frame 1305 to the resolution of the second enhancement layer.

  Further, each frame 1302, 1304, 1306, 1308 represents a frame of each layer at time t + δ. A frame 1302 represents a frame of the base layer at time t + δ. A frame 1306 represents a frame of the first enhancement layer at time t + δ. A frame 1304 represents a frame obtained by enlarging the decoded image of the frame 1302 to the resolution of the first enhancement layer. Frame 1310 represents the frame of the second enhancement layer at time t + δ. A frame 1308 represents a frame obtained by enlarging the decoded image of the frame 1306 to the resolution of the second enhancement layer.

  Hereinafter, the tile 5 of each frame (frames 1305, 1306, 1309, and 1310) of the enhancement layer will be described as an MCTS tile. In FIG. 13, a thick frame represents a tile belonging to the MCTS or its corresponding position.

  In FIG. 13, in order to decode the MCTS (tile 5) of the frame 1310 of the second enhancement layer, the tile 5 of the frame 1306 of the first enhancement layer needs to be decoded. Further, in order to decode the tile 5 of the frame 1306 of the first enhancement layer, the tile 0 of the frame 1302 of the base layer needs to be decoded. Further, in order to decode the tile 0 of the frame 1302 of the base layer, it is necessary to perform inter-frame prediction with reference to the frame 1301, and it is necessary to decode all the tiles of the frame 1301.

  That is, in the conventional technique, when decoding the MCTS of the second enhancement layer at time t + δ, a region other than the region indicating the position of the tile 5 of the frame 1302 of the base layer at time t (dotted line portion of the frame 1302) is decoded. There is a need. Therefore, in hierarchical encoding, when a predetermined tile is encoded and decoded using MCTS or the like, it is not possible to independently encode and decode only the tile corresponding to the position of the MCTS. There is.

  The present invention has been made to solve the above-described problem, and in hierarchical coding, it is possible to independently encode and decode a predetermined tile set as an MCTS without depending on other tiles. The purpose is that. Hereinafter, tiles that can be encoded and decoded independently like each tile included in the MCTS are referred to as independent tiles, and a collection of independent tiles such as MCTS is referred to as an independent tile set.

  The image encoding apparatus of the present invention has the following configuration, for example. That is, an image encoding apparatus that hierarchically encodes an image constituting a moving image in a plurality of layers, the generation unit generating a second image having a layer different from the first image, and the first image Encoding means for encoding the first area in the second image and the second area corresponding to the first area in the second image, and information encoding means, wherein the first area is The first image is a region that is restricted to refer to other than the first region, and the first region is encoded with reference to at least a partial region of the second image. The encoding unit encodes the first region with reference to only the second region in the region of the second image, and converts the second region into the second region. And encoding the information without referring to the area other than the second area. Encodes the information indicating that the reference to the second region outside when referring to the second image in the first region is limited.

  The image decoding apparatus of the present invention has the following configuration, for example. That is, an image decoding apparatus that decodes encoded data generated by hierarchically encoding an image constituting a moving image in a plurality of layers, the first region in the first image, the first image, Is a decoding means for decoding the first region and the second region corresponding to the second region in the second image having different layers, and the second region when the second image is referred to in the first region. Information decoding means for decoding information indicating that referring to the outside is restricted, and the first region is restricted from referring to other than the first region in the first image And when the first area is an area that is decoded with reference to at least a partial area of the second image, the decoding means determines the first area according to the information. , In the region of the second image, the second Decoded with reference to only a region, the second region is decoded without reference to other than the second region of the second image.

  The present invention makes it possible to set tiles that can be encoded and decoded independently in hierarchical encoding.

1 is a block diagram illustrating a configuration of an image encoding device 100 according to a first embodiment. A diagram showing an example of a tile configuration 7 is a flowchart illustrating image encoding processing in the image encoding device 100 according to the first embodiment. 1 is a block diagram illustrating a configuration of an image encoding device 400 according to Embodiment 1. FIG. 7 is a flowchart illustrating image encoding processing in the image encoding device 400 according to the first embodiment. FIG. 6 is a block diagram illustrating a configuration of an image display apparatus 600 according to the second embodiment. A block diagram showing composition of picture decoding part 605 in Embodiment 2. The flowchart showing the image decoding process in the image decoding part 605 of Embodiment 2. The block diagram which shows another structure of the image decoding part 605 apparatus in Embodiment 2. FIG. The flowchart showing the image decoding process in another form of the image decoding part 605 of Embodiment 2. The block diagram which shows the structural example of the hardware of the computer applicable to the image coding apparatus of this invention, or an image decoding apparatus. The figure showing the syntax about vui_parameters of a bit stream The figure which shows an example of the prior art example of the structure of a tile

  Hereinafter, the present invention will be described in detail based on the preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<Embodiment 1>
Hereinafter, the outline of each processing unit constituting the image coding apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing an image encoding device 100 of the present embodiment.

  Reference numeral 101 in FIG. 1 denotes a terminal (input means) for inputting an image (input image). The input image is input frame by frame. Reference numeral 102 denotes a tile setting unit that determines the number of tile divisions in the vertical and horizontal directions within one frame and the position of each tile. Further, the tile setting unit 102 determines whether or not to encode any one of the divided tiles as an independent tile. Hereinafter, the information set by the tile setting unit 102 and indicating the number of horizontal tile divisions, the number of vertical tile divisions, and the position of division is referred to as tile division information. The tile division information is described in Non-Patent Document 1 in the description part of Picture Parameter Set (PPS), which is the header data of a picture, so the description thereof is omitted here.

  Here, an example of tile division in the present embodiment will be described with reference to FIG. In FIG. 2 of the present embodiment, one frame is 4K2K (horizontal direction 4096 pixels × vertical direction 2160 pixels). Hereinafter, in this embodiment, 4096 pixels in the horizontal direction × 2160 pixels in the vertical direction will be referred to as 4096 × 2160 pixels. The same applies even if the number of pixels changes. Furthermore, 201 to 206 in FIG. 2 each represent a frame. Each of the frames 201 to 206 is composed of 12 tiles having tile numbers 0 to 11 by dividing the frame into four in the horizontal direction and three in the vertical direction. That is, the size of one tile is 1024 × 720 pixels. However, the number of divisions is not limited to this. Furthermore, the tile 5 and the tile 6 indicated by the thick frames in the frames 201 to 206 shown in FIG. 2 are respectively independent tiles, and an area composed of the tiles 5 and 6 is an independent tile set. Moreover, the thin frames in the frames 201 to 206 shown in FIG. 2 represent the boundaries of the tiles. Moreover, the thick frame in the enlarged image shown in FIG. 2 represents the position corresponding to these independent tile sets. Further, as is apparent from FIG. 2, in each layer, the number of tile divisions in the horizontal and vertical directions and the relative position of each tile are the same.

  A frame 201 in FIG. 2 represents the frame of the base layer input at time t. A frame 202 represents a frame of the base layer input at time t + δ. At time t + δ, the frame 201 has been encoded and locally decoded (inverse quantization and inverse transformation), and when the frame 202 is encoded, the locally decoded frame 201 can be used as a reference frame.

  The frame 203 is an enlarged image obtained by generating a reconstructed image by performing local decoding after encoding the frame 201 and further expanding the reconstructed image to a size equivalent to that of the enhancement layer. The frame 204 is an enlarged image obtained by encoding the frame 202, generating a reconstructed image by performing local decoding, and further expanding the reconstructed image to a size equivalent to that of the enhancement layer.

  A frame 205 represents an enhancement layer frame input at time t. A frame 206 represents an enhancement layer frame input at time t + δ.

  Returning to the description of each processing unit in FIG. Hereinafter, the frame at time t + δ will be described as a frame to be encoded.

  The tile setting unit 102 generates an independent tile flag representing information on whether or not an independent tile is included in sequence units. The tile setting unit 102 sets the value of the independent tile flag to 1 when the encoding target frame includes an independent tile, and sets the value of the independent tile flag to 0 when the encoding target frame does not include the independent tile. To do. Further, when an independent tile is included in the encoding target frame (the value of the independent tile flag is 1), the tile setting unit 102 generates independent tile position information indicating the position of the independent tile. In general, the independent tile position information is represented by a tile number in the image, but the present invention is not limited to this. Then, the tile setting unit 102 outputs the generated independent tile flag and independent tile position information to the subsequent stage as tile division information. In the present embodiment, the tile division information output from the tile setting unit 102 is input to the enhancement layer division unit 104, the base layer division unit 105, the independent tile determination unit 106, and the header encoding unit 114.

  Reference numeral 103 denotes a reduction unit. The reduction unit 103 reduces the input image input from the terminal 101 using a predetermined filter or the like, and generates a reduced image (base layer image) with reduced resolution.

  Reference numeral 104 denotes an enhancement layer dividing unit. The enhancement layer dividing unit 104 sets an input image input from the terminal 101 as an enhancement layer image (enhancement layer image), and based on the tile division information output by the tile setting unit 102, the enhancement layer image is converted into one or more enhancement layer images. Divide into tiles. Here, the enhancement layer dividing unit 104 divides the input frame 206 into 12 tiles 0 to 11 as shown in FIG. Further, the enhancement layer dividing unit 104 outputs each divided tile to the subsequent stage in the order of tile numbers (in the order of 0, 1, 2,..., 11).

  Reference numeral 114 denotes a header encoding unit. Header code data in units of sequences and pictures is generated. In particular, the header encoding unit 114 receives the independent tile flag and the independent tile position information generated by the tile setting unit 102, generates MCTS SEI (SEI message), and encodes VUI parameters (vui_parameters).

  Reference numeral 105 denotes a base layer dividing unit. The base layer division unit 105 divides the base layer image generated by the reduction unit 103 into one or more tiles based on the tile division information output by the tile setting unit 102. That is, the base layer division unit 105 divides the base layer image into tiles such that the positions of the tiles based on the tile division information are relatively equal in the base layer image generated by the reduction unit 103. . In the present embodiment, the base layer dividing unit 105 divides the input frame 202 into 12 tiles 0 to 11 as shown in FIG. Further, the base layer division unit 105 outputs the divided tiles to the subsequent stage in the order of tile numbers. Further, the base layer division unit 105 notifies the independent tile determination unit 106 of the number of the tile to be output (tile to be encoded).

  Reference numeral 106 denotes an independent tile determination unit that determines whether or not an encoding target tile (encoding target tile) is an independent tile. The independent tile determination unit 106 determines whether the encoding target tile is based on the independent tile flag and the independent tile position information generated by the tile setting unit 102 and the encoding target tile number input from the base layer division unit 105. Determine whether it is an independent tile. Here, when the independent tile flag is 1, the position of the independent tile is the tile 5 according to the independent tile position information, and the encoding target tile is the tile 5, the independent tile determination unit 106 determines that the encoding target tile is It can be determined that the tile is an independent tile. Furthermore, the independent tile determination unit 106 outputs the determination result to the subsequent stage as an independent tile encoding flag. Here, the independent tile determination unit 106 sets the value of the independent tile encoding flag to 1 when the encoding target tile is an independent tile, and sets the independent tile encoding flag when the encoding target tile is not an independent tile. The value of is assumed to be 0.

  Reference numeral 107 denotes a base layer encoding unit that encodes the encoding target tile image of the base layer image input from the base layer dividing unit 105. The base layer encoding unit 107 encodes the encoding target tile based on the independent tile encoding flag input from the independent tile determination unit 106, and generates base layer code data.

  Here, the encoding process in the base layer encoding unit 107 when the independent tile encoding flag indicates that the encoding target tile is an independent tile will be described. In this case, the base layer encoding unit 107 performs prediction and encoding with reference to only a pixel at a position relatively equal to the position of the independent tile set including the encoding target tile in the reconstructed image of the base layer that has been locally decoded. Do. Further, referring to FIG. 2 as an example, when the tile 5 of the frame 202 is to be encoded, the base layer encoding unit 107 performs prediction with reference to only the tile 5 and the tile 6 in the independent tile set of the frame 201. And encoding. On the other hand, when the independent tile encoding flag indicates that the encoding target tile is not an independent tile, the base layer encoding unit 107 refers to all pixels of the reconstructed image of the base layer that has been locally decoded, and performs prediction and prediction. Encode errors and so on. Referring to FIG. 2, when the tile 2 of the frame 202 is to be encoded, the base layer encoding unit 107 performs prediction and encoding with reference to all the tiles (tiles 0 to 11) of the frame 201. Do.

  Furthermore, the base layer encoding unit 107 outputs the prediction mode used for prediction, the prediction error generated by the prediction, the base layer code data generated by encoding the prediction error, and the like to the subsequent stage.

  Reference numeral 108 denotes a base layer reconstruction unit that receives the coefficients (prediction mode and prediction error) generated by the base layer coding unit 107 and locally decodes the prediction error to generate a reconstructed image of the base layer. Furthermore, the base layer reconstruction unit 108 holds the generated reconstructed image. This is because the base layer encoding unit 107 and the enhancement layer encoding unit 112 perform prediction using the reconstructed image.

  Reference numeral 109 denotes an enlargement unit that enlarges the reconstructed image of the base layer to the size of the enhancement layer. In FIG. 2, the enlargement unit 109 enlarges each reconstructed image of the frame 201 and the frame 202 to generate a frame 203 and a frame 204.

  Reference numeral 112 denotes an enhancement layer encoding unit that encodes a tile image input from the enhancement layer dividing unit 104. The enhancement layer encoding unit 112 selects a reference image based on the independent tile encoding flag input from the independent tile determination unit 106, encodes the encoding target tile, and generates enhancement layer code data.

  Here, when the independent tile encoding flag is 1 (the encoding target tile is an independent tile), the enhancement layer encoding unit 112 expands the reconstructed image of the base layer that has been locally decoded, and local decoding. Reference is made to the reconstructed image of the completed enhancement layer. Then, the enhancement layer encoding unit 112 performs prediction and encoding with reference to images included in the independent tile set of each image of the enlarged image and the reconstructed image. Further, referring to FIG. 2 as an example, when the tile 5 of the frame 206 is to be encoded, the enhancement layer encoding unit 112 performs local decoding among the tile 5 and the tile 6 of the frame 204 and the tile 5 of the frame 206. Prediction and encoding are performed with reference to the completed reconstructed image. On the other hand, when the independent tile encoding flag is 0 (the encoding target tile is not an independent tile), the enhancement layer encoding unit 112 performs the locally decoded base layer enlarged image and the locally decoded enhancement layer reconstructed image. To make predictions without limiting to independent tiles. Then, the enhancement layer encoding unit 112 encodes a prediction error or the like generated by prediction.

  Further, similar to the base layer encoding unit 107, the enhancement layer encoding unit 112, the prediction mode used for prediction, the prediction error generated by the prediction, and the enhancement layer code generated by encoding the prediction error Output data etc. to the subsequent stage.

  Reference numeral 113 denotes an enhancement layer reconstruction unit that performs local decoding using coefficients (prediction mode and prediction error) generated in the middle of coding by the enhancement layer coding unit 112 and generates a reconstructed image of the enhancement layer. . Furthermore, the enhancement layer reconstruction unit 113 holds the generated reconstructed image for use in the encoding process in the enhancement layer encoding unit 112.

  110 integrates the base layer code data generated by the base layer encoding unit 107, the enhancement layer code data generated by the enhancement layer encoding unit 112, and the header code data generated by the header encoding unit 114, and a bit stream Is an integration unit that generates Reference numeral 111 denotes a terminal that outputs the bitstream generated by the integration unit 110 to the outside.

  The overall control unit 115 controls each processing unit in the image encoding device and transmits parameters between the processing units. In FIG. 1, the connection between the overall control unit 115 and each processing unit in the image coding apparatus is omitted. The overall control unit 115 can control each processing unit in the image coding apparatus and read / write parameters between the processing units through either a parameter signal line or a register bus. In the present embodiment, the overall control unit 115 in FIG. 1 is installed in the image encoding apparatus, but the present invention is not limited to this. That is, the overall control unit 115 is installed outside the image encoding device, and controls each processing unit in the image encoding device and reads / writes parameters between the processing units using either a parameter signal line or a register bus. You may go through.

  An image encoding operation in the image encoding apparatus 100 described above will be described below with reference to the flowchart shown in FIG.

  In step S301, the image coding apparatus 100 acquires the number of layers of layer coding instructed by the user. In the present embodiment, it is assumed that the enhancement layer is one layer, and hierarchical coding of two layers (base layer and one enhancement layer) is performed as a whole.

  In step S302, the tile setting unit 102 determines the number of tile divisions and the division positions in the encoding target frame, and determines whether or not any tile in the encoding target frame is an independent tile. To decide. In this embodiment, the tile 5 and the tile 6 are independent tiles, and the tile 5 and the tile 6 are combined to form one independent tile set. Therefore, in the present embodiment, the independent tile determination unit 106 sets the independent tile flag to 1. Incidentally, when an independent tile is not included in the encoding target frame, the independent tile determination unit 106 sets the independent tile flag to 0. Further, the independent tile determination unit 106 inputs the determined independent tile flag to the enhancement layer division unit 104, the base layer division unit 105, the independent tile determination unit 106, and the header encoding unit 114.

  In step S303, the header encoding unit 114 determines the independent tile flag input from the independent tile determination unit 106. If the header encoding unit 114 determines that the independent tile flag is 1, the process proceeds to step S304. If the header encoding unit 114 determines that the independent tile flag is 0, the process proceeds to step S305.

  In step S304, the header encoding unit 114 sets the tile_boundaries_aligned_flag code of vui_parameters representing the matching information of the positions of the tiles to 1. Note that the tile_boundaries_aligned_flag code of the vui_parameters is obtained by encoding coincidence information indicating whether or not the relative positions of the tiles coincide between the layers.

  In step S305, the header encoding unit 114 encodes video_parameter_set, which is one of the sequence headers. The video_parameter_set code includes a vps_max_layers_minus1 code that represents the number of layers of hierarchical coding. In this embodiment, vps_max_layers_minus1 is 1. Subsequently, the header encoding unit 114 encodes a Sequence parameter set (described in 7.3.2.2 in Non-Patent Document 1). The sequence parameter set code also includes vui_parameters. The vui_parameters include the tile_boundaries_aligned_flag code set in step S304. The integration unit 110 inputs these code data (video_parameter_set code and Sequence parameter set code), and generates a bitstream. Further, the integration unit 110 outputs the generated bit stream to the outside of the image encoding device 100 via the terminal 111.

  In step S306, the header encoding unit 114 encodes a picture parameter set (described in 7.4.3.3 in Non-Patent Document 1) that is a picture header. The integration unit 110 receives the code data (Picture parameter set code) of the picture header, and generates a bit stream. Further, the integration unit 110 outputs the generated bit stream to the outside of the image encoding device 100 via the terminal 111.

  In step S307, the header encoding unit 114 determines the independent tile flag input from the independent tile determination unit 106. If the header encoding unit 114 determines that the independent tile flag is 1, the process proceeds to step S308. If the header encoding unit 114 determines that the independent tile flag is 0, the process proceeds to step S309.

  In step S308, since the encoding target sequence includes independent tiles, the header encoding unit 114 encodes MCTS SEI. The MCTS SEI code is as described in Chapter 2 of Non-Patent Document 2. In this embodiment, since one independent tile set is included in one frame, the num_sets_in_message_minus1 code is 0. The mcts_id code is 0. Further, the num_tile_rects_in_set_minus1 code is 1. The num_tile_rects_in_set_minus1 code represents the number of independent tiles belonging to the MCTS. In the present embodiment, two tiles of the tile 5 and the tile 6 are included as independent tiles in the independent tile set, so that the value of the num_tile_rects_in_set_minus1 code is 1. Further, the top_left_tile_index code and the bottom_right_tile_index code represent the positions of independent tiles. In the present embodiment, the former value is 5 and the latter value is 6. The header encoding unit 114 encodes each piece of header information as described above to generate a MCTS SEI code. Further, the integration unit 110 inputs the MCTS SEI code generated by the header encoding unit 114 to generate a bit stream, and outputs the bit stream to the outside of the image encoding device 100 via the terminal 111.

  In step S309, the reduction unit 103 reduces the input image and generates a base layer image. In this embodiment, since the enhancement layer is one layer, the base layer is generated by the reduction unit 103, but the present invention is not limited to this. In the case of hierarchical encoding with two or more enhancement layers (three or more layers in total), a plurality of reduction units 103 may be provided, or one reduction unit 103 may generate images of the required number of layers.

  In step S310, the base layer dividing unit 105 extracts base layer tile images to be encoded in the order of tile numbers from the upper left of the image. The base layer dividing unit 105 outputs the extracted base layer tile image to the base layer encoding unit 107.

  In step S <b> 311, the independent tile determination unit 106 inputs the tile number of the encoding target tile from the base layer division unit 105. Further, the independent tile determination unit 106 inputs the independent tile position information of the encoding target tile from the tile setting unit 102. In this embodiment, the independent tile position information is 5 and 6. The independent tile determination unit 106 compares the input tile number of the encoding target tile with the tile number of the independent tile position information. When the tile number of the encoding target tile matches the tile number of the independent tile position information, the independent tile determination unit 106 determines that the encoding target tile is an independent tile, sets the independent tile encoding flag to 1, Proceed to step S312. On the other hand, if the tile number of the encoding target tile does not match the tile number of the independent tile position information, the independent tile determination unit 106 determines that the encoding target tile is not an independent tile, and sets the independent tile encoding flag to 0. And go to Step S313.

  In step S312, the encoding target tile is an independent tile in the base layer encoding target frame. Therefore, the base layer encoding unit 107 refers to the reconstructed image included in the independent tile set at a position relatively equal to the position of the encoding target tile in another frame of the base layer that has been locally decoded. Perform inter-frame prediction and encoding. In addition, the base layer encoding unit 107 performs intra prediction and encoding with reference to the reconstructed image that has been locally decoded in the encoding target tile of the encoding target frame. A case where the tile 5 of the frame 202 is encoded in FIG. 2 will be described. The base layer encoding unit 107 performs prediction and encoding with reference to the locally decoded reconstructed images of the tile 5 and the tile 6 of the frame 201 and the tile 5 of the frame 202 stored in the base layer reconstruction unit 108. I do. Furthermore, the base layer encoding unit 107 outputs the code data of the encoding target tile of the base layer obtained by encoding to the integrating unit 110 as base layer code data. The integration unit 110 integrates the base layer code data output from the base layer encoding unit 107 and the other code data output from the header encoding unit 114 and the enhancement layer encoding unit 112, and generates a bitstream. . Then, the integration unit 110 outputs the generated bit stream via the terminal 111. In addition, the base layer reconstruction unit 108 sequentially generates a reconstructed image of the base layer using the coefficients (prediction mode and prediction residual) generated in the middle of the encoding by the base layer encoding unit 107, Hold.

  In step S313, the encoding target tile is not an independent tile in the base layer encoding target frame. For this reason, the base layer encoding unit 107 performs inter-frame prediction and encoding of the encoding target tile with reference to the entire image of another frame of the base layer that has been locally decoded. In FIG. 2, when encoding tile 5 of frame 202, base layer encoding section 107 performs local decoding of all tiles of frame 201 and tile 5 of frame 202 stored in base layer reconstruction section 108. Prediction and encoding are performed with reference to the already reconstructed image. Furthermore, the base layer encoding unit 107 outputs the generated base layer code data to the integration unit 110. Similarly to the description in step S <b> 312, the integrating unit 110 integrates the base layer code data and other code data to generate a bit stream, and outputs the bit stream via the terminal 111. Furthermore, the base layer reconstructing unit 108 sequentially generates and retains base layer reconstructed images using the coefficients generated in the middle of the encoding by the base layer encoding unit 107.

  In step S314, overall control unit 115 determines whether all tiles of the base layer have been encoded. When it is determined that the encoding process for all the tiles of the base layer is not completed (NO in step S314), the process returns to step S310, and the base layer dividing unit 105 extracts and outputs the tile having the next tile number, and the process To continue. On the other hand, when it is determined that the encoding process for all the tile layer images has been completed (YES in step S314), the process proceeds to step S315.

  In step S315, the enhancement layer dividing unit 104 extracts tile layer images to be encoded in the order of tile numbers from the upper left of the image. The enhancement layer dividing unit 104 outputs the extracted enhancement layer tile image to the enhancement layer encoding unit 112.

  In step S316, the independent tile determination unit 106 compares the input tile number of the encoding target tile with the tile number of the independent tile position information, similarly to the processing in step S311. When the tile number of the encoding target tile matches the tile number of the independent tile position information, the independent tile determination unit 106 determines that the encoding target tile is an independent tile, sets the independent tile encoding flag to 1, The process proceeds to step S317. On the other hand, if the tile number of the encoding target tile does not match the tile number of the independent tile position information, the independent tile determination unit 106 determines that the encoding target tile is not an independent tile, and sets the independent tile encoding flag to 0. And go to step S319.

  In step S317, the encoding target tile is an independent tile in the encoding target frame of the enhancement layer. For this reason, the expansion unit 109 is included in the independent tile set at a position relatively equal to the position of the encoding target tile from the reconstructed image of the base layer that has been locally decoded and stored in the base layer reconstruction unit 108. Enter the reconstructed image. The enlargement unit 109 uses only the input reconstructed image to generate an enlarged image by filtering or the like, and outputs the enlarged image to the enhancement layer encoding unit 112.

  In step S318, the enhancement layer encoding unit 112 predicts and encodes the image of the encoding target tile input from the enhancement layer dividing unit 104 using the reconstructed image of the base layer that has been locally decoded as a reference target. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image generated in step S317. Also, the enhancement layer encoding unit 112 is a reconstructed image of an independent tile set at a position relatively equal to the position of the encoding target tile among the locally decoded enhancement layers stored in the enhancement layer reconstruction unit 113. As a reference target, inter-frame prediction of the encoding target tile is performed. Furthermore, the enhancement layer encoding unit 112 performs intra prediction using a locally decoded reconstructed image in the encoding target tile as a reference target. The enhancement layer encoding unit 112 encodes information related to prediction (such as a motion vector obtained by inter-frame prediction) and a prediction error obtained by these predictions. Furthermore, the enhancement layer reconfiguration unit 113 sequentially generates reconstructed images of the enhancement layer using the coefficients (prediction mode and prediction residual) generated during the encoding by the enhancement layer encoding unit 112, Hold.

  In step S319, the encoding target tile is not an independent tile in the enhancement target layer encoding target frame. For this reason, the enlarging unit 109 generates an enlarged image by performing filtering or the like using the entire reconstructed image of the base layer stored in the base layer reconstructing unit 108, and the enlarged image is encoded with an enhancement layer code. To the conversion unit 112.

  In step S320, enhancement layer encoding section 112 encodes the encoding target tile image input from enhancement layer dividing section 104 with reference to the base layer locally decoded reconstructed image. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image generated in step S319. Also, the enhancement layer encoding unit 112 refers to the reconstructed image of the enhancement layer that has been locally decoded and stored in the enhancement layer reconstruction unit 113, and performs interframe prediction of the encoding target tile. Furthermore, the enhancement layer encoding unit 112 performs intra prediction of the encoding target tile with reference to the locally decoded reconstructed image in the encoding target tile. The enhancement layer encoding unit 112 encodes information and prediction error related to prediction obtained by these predictions. Furthermore, the enhancement layer reconstruction unit 113 sequentially generates and holds enhancement layer reconstructed images using the coefficients and the like generated during the encoding by the enhancement layer encoding unit 112.

  In step S321, overall control unit 115 determines whether or not all tiles of the enhancement layer have been encoded. If it is determined that all the tiles of the enhancement layer have not been encoded (NO in step S321), the process returns to step S315, and the enhancement layer dividing unit 104 extracts and outputs the tile having the next tile number, and performs the process. continue. On the other hand, if it is determined that the image encoding process for all tiles in the enhancement layer has been completed (YES in step S321), the process proceeds to step S322.

  In step S322, overall control unit 115 determines whether or not the encoding processing for all the frames included in the sequence input from terminal 101 has been completed. If there is a frame that has not been encoded (NO in step S322), the process proceeds to step S309 to process the next frame. If there is no frame that has not been encoded (YES in step S322), the encoding process is terminated.

  With the above configuration and operation, when using independent tiles and independent tile sets, the relative positions of the tiles of the enhancement layer and the base layer can be matched. That is, the tiles included in the independent tile set set in the base layer are set so as to be included in the independent tile set at a position relatively equal to the independent tile set in each expansion layer. Thereby, in any hierarchy of hierarchical encoding, the pixel referred for prediction and decoding of an independent tile can be restrict | limited, and a prediction process can be sped up. In particular, by setting the attention area etc. as independent tiles, independent tiles can be encoded independently without referring to other tiles from the base layer to the extended layer, so the necessary parts are processed faster than before. It becomes possible.

  In this embodiment, as shown in FIG. 2, an example is shown in which only a frame temporally prior to the encoding target frame is predicted and encoded as a reference frame, but the present invention is not limited to this. In other words, it is clear from the above description that the same reference is made in the case of prediction and encoding with reference to a plurality of frames.

  In the present embodiment, the image coding apparatus 100 using the reduction unit 103 and the enlargement unit 109 has been described. However, the present invention is not limited to this. That is, the reduction unit 103 and the enlargement unit 109 may be omitted. Alternatively, the reduction parameter and the enlargement factor may be set to 1, and the quantization parameter set by the enhancement layer encoding unit 112 may be made smaller than the quantization parameter set by the base layer encoding unit 107. This makes it possible to perform SNR hierarchical coding.

  In the present embodiment, the enlarged image referred to when predicting the tiles of the independent tile set of the enhancement layer is generated using only the tile image of the basic layer at a position relatively equal to the independent tile set. However, the present invention is not limited to this. That is, as in step S319, the pixels around the independent tile of the base layer may be used as a reference target.

  Further, in the present embodiment, it has been described that the hierarchical encoding of the basic layer and the enhancement layer of one hierarchy (two hierarchical encoding as a whole) is performed, but the present invention is not limited to this, and three hierarchical layers as a whole. The above hierarchical encoding may be used. In this case, the reduction unit 103, the enhancement layer division unit 104, the enhancement layer encoding unit 112, the enhancement layer reconstruction unit 113, and the enlargement unit 109 are set as one set, and the set is provided for the number of enhancement layer hierarchies. , Can accommodate more hierarchies. Also, as shown in FIG. 4, the enhancement layer encoding unit 112, the enhancement layer reconstruction unit 413, the enlargement unit 409, and the reduction unit 403 are provided one by one, and are used for encoding each enhancement layer. It doesn't matter.

  FIG. 4 is an image encoding apparatus capable of encoding a plurality of enhancement layers, and includes an enhancement layer encoding unit 112, an enhancement layer reconstruction unit 413, an enlargement unit 409, and a reduction unit 403 one by one. It is a block diagram of an image coding apparatus. In FIG. 4, the same numbers are assigned to components that perform the same functions as the processing units of the image encoding device 100 in FIG. 1, and descriptions thereof are omitted. Reference numeral 401 denotes a layer number setting unit for setting the number of layers for layer encoding. Reference numeral 403 denotes a reduction unit. While the reduction unit 103 in FIG. 1 reduces the input image input from the terminal 101 to generate one reduced image, the reduction unit 403 uses the input image based on the number of hierarchies input from the hierarchy number setting unit 401. Are reduced to generate reduced images of a plurality of hierarchies. Reference numeral 402 denotes a frame memory, which stores the reduced image of each layer generated by the reduction unit 403. Reference numeral 409 denotes an enlarged portion. The enlargement unit 109 in FIG. 1 enlarges the reconstructed image of the base layer to the size of the extension layer to generate one enlarged image, whereas the enlargement unit 109 is based on the number of hierarchies input from the hierarchy number setting unit 401. The reconstructed image is enlarged to generate enlarged images having a plurality of different resolution layers. Reference numeral 413 denotes an enhancement layer reconstruction unit. The enhancement layer reconstruction unit 413 receives the number of layers from the layer number setting unit 401, generates a reconstruction image of the enhancement layer using the coefficients generated by the enhancement layer encoding unit 112, and the reconstructed image The result is output to expansion section 409 and enhancement layer encoding section 112. Reference numeral 410 denotes an integration unit that inputs the number of layers from the layer number setting unit 401 and integrates code data for the number of layers into a bitstream.

  The operation of each processing unit when encoding is performed using the image encoding device 400 illustrated in FIG. 4 will be described below with reference to the flowchart illustrated in FIG. FIG. 5 shows only the part changed from step S309 to step S320 in FIG. In FIG. 5, steps having the same functions as those in FIG. 3 are given the same numbers as in FIG. Further, the following description will be made assuming that the number of layers setting unit 401 sets the number of layers to 3 in step S301 of FIG. In the present invention, the number of hierarchies is not particularly limited. In step S305, it is assumed that header code data is generated with the vps_max_layers_minus1 code set to 2.

  In step S501, the reduction unit 403 generates reduced images for the number of layers of one frame. In the present embodiment, since the number of hierarchies is set to 3 in step S301, the reduction unit 403 generates one base layer image and two enhancement layer images. That is, the reduction unit 403 generates an extended first layer (first enhancement layer) image in which the input image is halved in the vertical and horizontal directions, and a base layer image in which the first enhancement layer image is further halved in the vertical and horizontal directions. Here, the reduction unit 403 sets the input resolution image as an extended second layer (second extended layer) image. Further, the reduction unit 403 outputs the base layer image, the first enhancement layer image, and the second enhancement layer image to the frame memory 402, respectively.

  In steps S312 to S314, the overall control unit 115 encodes the base layer image output from the frame memory 402 as described above. The base layer reconstruction unit 108 locally decodes the encoded image to generate a reconstructed image, and holds this.

  In step S502, the layer number setting unit 401 sets the base layer encoded in steps S312 to S313 or the enhancement layer of the layer encoded in steps S518 to S520 described later as an upper layer. Further, the hierarchy number setting unit 401 sets the enhancement layer to be encoded following that layer as a lower layer. Here, first, the base layer encoded in steps S312 to S313 is set as an upper layer, and the first enhancement layer is set as a lower layer.

  In step S515, the enhancement layer dividing unit 104 extracts the tile images of the enhancement layer to be coded in order of tile numbers from the upper left of the image of the layer to be coded. The enhancement layer dividing unit 104 outputs the extracted enhancement layer tile image to the enhancement layer encoding unit 112. Here, the image of the encoding target tile in the first enhancement layer image is extracted and input to the enhancement layer encoding unit 112.

  In step S517, the encoding target tile is an independent tile in the encoding target frame. For this reason, the enlargement unit 409 determines the independent tile set at a position relatively equal to the position of the encoding target tile from the upper layer reconstructed image stored in the base layer reconstruction unit 108 or the enhancement layer reconstruction unit 413. The reconstructed image included in is input. The enlargement unit 409 uses only the input reconstructed image to generate an enlarged image by filtering or the like, and inputs the enlarged image to the enhancement layer encoding unit 112. Here, the enlargement unit 409 generates an enlarged image from the reconstructed image stored in the base layer reconstruction unit 108 and inputs the enlarged image to the enhancement layer encoding unit 112.

  In step S518, enhancement layer encoding section 112 predicts and encodes the encoding target tile image input from enhancement layer dividing section 104 with reference to the locally decoded reconstructed image. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image generated in step S517. Also, the enhancement layer encoding unit 112 reconstructs an independent tile set at a position relatively equal to the position of the encoding target tile in another frame of the enhancement layer that has been locally decoded and stored in the enhancement layer reconstruction unit 413. Inter-frame prediction is performed with reference to the constituent images. Furthermore, the enhancement layer encoding unit 112 performs intra prediction with reference to the locally decoded reconstructed image in the encoding target tile. The enhancement layer encoding unit 112 encodes information related to prediction (such as a motion vector obtained by inter-frame prediction) and a prediction error obtained by these predictions. Furthermore, the enhancement layer reconfiguration unit 413 sequentially generates reconstructed images of the enhancement layer using coefficients (prediction mode and prediction residual) generated in the middle of the encoding by the enhancement layer encoding unit 112, Hold.

  In step S519, the encoding target tile is not an independent tile in the encoding target frame. Therefore, the enlarging unit 409 displays the entire base layer reconstructed image stored in the base layer reconstructing unit 108 or the entire upper layer reconstructed image stored in the enhancement layer reconstructing unit 413. The image is enlarged by filtering or the like to generate an enlarged image. Further, the enlargement unit 409 outputs the generated enlarged image to the enhancement layer encoding unit 112. Here, the enlargement unit 409 generates an enlarged image from the reconstructed image stored in the base layer reconstruction unit 108.

  In step S520, enhancement layer encoding section 112 encodes the encoding target tile image input from enhancement layer dividing section 104 with reference to the locally decoded reconstructed image. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image generated in step S519. Also, the enhancement layer encoding unit 112 refers to the reconstructed image of the enhancement layer that has been locally decoded and stored in the enhancement layer reconstruction unit 413, and performs interframe prediction of the encoding target tile. Furthermore, the enhancement layer encoding unit 112 performs intra prediction of the encoding target tile with reference to the locally decoded reconstructed image in the encoding target tile. The enhancement layer encoding unit 112 encodes information and prediction error related to prediction obtained by these predictions. Furthermore, the enhancement layer reconstruction unit 413 sequentially generates and holds enhancement layer reconstructed images using the coefficients and the like generated during the encoding by the enhancement layer encoding unit 112.

  In step S503, overall control unit 115 determines whether or not encoding has been completed for all layers set by layer number setting unit 401. When it is determined that the encoding process for all the tiles has not been completed (NO in step S521), the process returns to step S502, and the hierarchy number setting unit 401 sets the next hierarchy as a lower layer and continues the process. On the other hand, when it is determined that the image encoding process for all tiles in the enhancement layer has been completed (YES in step S521), the process proceeds to step S523. Here, it is determined that the encoding of the second enhancement layer has not ended, and the process returns to step S502.

  In step S522, overall control unit 115 determines whether or not the encoding processing for all the frames included in the sequence input from terminal 101 has been completed. If there is a frame that has not been subjected to the encoding process (NO in step S522), the process proceeds to step S501 to process the next frame. If there is no frame that has not been subjected to the encoding process (YES in step S522), the encoding process ends.

  Hereinafter, the encoding process of the second enhancement layer image will be described. That is, in step S502, the layer number setting unit 401 sets the first enhancement layer layer encoded in steps S518 to S520 as an upper layer and the second enhancement layer as a lower layer. In step S515, enhancement layer dividing section 104 extracts an image of the encoding target tile in the second enhancement layer image, and inputs it to enhancement layer encoding section 112.

  In step S517, the encoding target tile is an independent tile in the encoding target frame. For this reason, the enlargement unit 409 converts an upper layer (first enhancement layer) reconstructed image stored in the enhancement layer reconstruction unit 413 into an independent tile set at a position relatively equal to the position of the encoding target tile. Input the reconstructed image to be included. The enlargement unit 409 uses only the reconstructed image of the input independent tile set to generate an enlarged image of a higher layer (first enhancement layer) by filtering or the like, and the enlarged image is an enhancement layer encoding unit 112. In step S518, enhancement layer encoding section 112 refers to the tiled image of the encoding target lower layer (second enhancement layer) input from enhancement layer dividing section 104, and the locally decoded reconstructed image. Predict and encode. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image of the upper layer (first enhancement layer) generated in step S517. Also, the enhancement layer encoding unit 112 is an independent tile set at a position relatively equal to the position of the encoding target tile in the locally decoded lower layer (second enhancement layer) stored in the enhancement layer reconstruction unit 413. The inter-frame prediction is performed with reference to the image. Furthermore, the enhancement layer encoding unit 112 performs intra prediction with reference to the locally decoded reconstructed image of the lower layer (second enhancement layer) in the encoding target tile. The enhancement layer encoding unit 112 encodes information related to prediction (such as a motion vector obtained by inter-frame prediction) and a prediction error obtained by these predictions. Furthermore, the enhancement layer reconfiguration unit 413 sequentially generates and holds a reconstructed image of the lower layer (second enhancement layer) using the coefficients and the like generated during the encoding by the enhancement layer encoding unit 112. .

  On the other hand, in step S519, the encoding target tile is not an independent tile in the encoding target frame. Therefore, the enlargement unit 409 enlarges the upper layer (first enhancement layer) by filtering or the like using the reconstructed image of the upper enhancement layer (first enhancement layer) stored in the enhancement layer reconstruction unit 413. An enlarged image of is generated. Further, the enlargement unit 409 outputs the generated enlarged image to the enhancement layer encoding unit 112.

  In step S520, enhancement layer encoding section 112 encodes the encoding target tile image input from enhancement layer dividing section 104 with reference to the locally decoded reconstructed image. That is, the enhancement layer encoding unit 112 performs inter-layer prediction with reference to the enlarged image of the upper layer (first enhancement layer) generated in step S519. Also, the enhancement layer encoding unit 112 performs inter-frame prediction with reference to the reconstructed image of the locally decoded lower layer (second enhancement layer hierarchy) stored in the enhancement layer reconstruction unit 413. Further, the enhancement layer encoding unit 112 performs intra prediction with reference to the locally decoded reconstructed image in the encoding target tile of the lower layer (second enhancement layer). The enhancement layer encoding unit 112 encodes information and prediction error related to prediction obtained by these predictions. Furthermore, the enhancement layer reconstruction unit 413 sequentially generates and holds the reconstructed raw image of the lower layer (second enhancement layer) using the coefficients and the like generated during the encoding by the enhancement layer encoding unit 112. To do.

  In step S503, the overall control unit 115 determines whether or not encoding has been completed for all the layers set by the number-of-layers setting unit 401. If it is determined that encoding has been completed, the overall control unit 115 proceeds to step S522. If it is determined that the process has not ended, the process returns to step S502. Here, since the encoding up to the second enhancement layer has been completed, the process proceeds to step S522. If the encoding of all the frames is completed in step S522, the encoding process is ended.

  With the above operation, even when there are a plurality of enhancement layers, it is possible to decode only the code data that requires the independent tile set and generate code data that can reproduce the decoded image only by referring to the minimum image.

  When the MCTS SEI code is present in the bitstream, the tile_boundaries_aligned_flag code of vui_parameters, which is tile position match information, is always set to 1. That is, when MCTS SEI code is present in the bitstream in vui_parameters, the tile_boundaries_aligned_flag code as code data can be omitted. If the MCTS SEI code is not in the bitstream, the value of the tile_boundaries_aligned_flag code is encoded, and the code data is included in the bitstream. If the MCTS SEI code is in the bitstream, the value of the tile_boundaries_aligned_flag code is not encoded, and a value of 1 is always set on the decoding side. This makes it possible to reduce redundant tile_boundaries_aligned_flag codes.

  Further, in hierarchical encoding, an important area is cut out and encoded by applying an independent tile set to the extracted area, thereby generating code data capable of reading out the important area at high speed.

<Embodiment 2>
Hereinafter, the outline of each processing unit constituting the image decoding apparatus according to the present embodiment will be described with reference to FIG. FIG. 6 is a block diagram showing an image display device 600 having the image decoding unit 605 of the present embodiment. In the present embodiment, the case where the bitstream generated in the first embodiment is decoded will be described as an example.

  Reference numeral 601 denotes an interface for inputting a bit stream by communication or the like. A storage unit 602 stores a bit stream input from the interface 601 and a bit stream recorded in advance. Reference numeral 603 denotes a display control unit that designates a display method for displaying the bitstream designated by the user. The display control unit 603 outputs the decoding hierarchy (layer) and the decoding area (display area) to the image decoding unit 605 as display control signals. In this embodiment, the hierarchy to be decoded is represented by the number of hierarchies, and the display area is represented by the position of the tile to be displayed. A selector 604 designates an input destination of an input bit stream. Reference numeral 605 denotes an image decoding unit according to the present embodiment. Reference numeral 606 denotes a display unit that displays the decoded image generated by the image decoding unit 605.

  Next, an image display operation in the image display apparatus 600 will be described below. A case will be described in which the display control unit 603 is instructed by the user to decode and display the base layer image of the bitstream. This corresponds to a case where a bit stream obtained by encoding an image captured by a monitoring camera or the like is input and the entire captured image is monitored. The interface 601 receives a bit stream (input bit stream) input in units of frames from a monitoring camera or the like, and outputs it to the storage unit 602 and the selector 604. The storage unit 602 records the input bit stream, and the selector 604 outputs the input bit stream to the image decoding unit 605 according to an instruction from the display control unit 603. The image decoding unit 605 receives information such as a layer to be displayed as a display control signal and a tile to be displayed from the display control unit 603. That is, since the display control unit 603 is instructed by the user to decode and display the base layer of the bitstream, the image decoding unit 605 includes information indicating that the layer to be decoded is the base layer, and all display areas are displayed. Information indicating that it is a tile is input.

  Hereinafter, the outline of each processing unit constituting the image decoding unit 605 according to the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram showing the image decoding unit 605 of the present embodiment.

  701 in FIG. 7 is a terminal for inputting the bit stream output from the selector 604. For ease of explanation, it is assumed that the bit stream is input with header data and code data for each frame. In the present embodiment, it is assumed that this frame-unit code data includes all the hierarchical code data constituting one frame, but the present invention is not limited to this, and is input in units such as slices. It doesn't matter. The data structure of the frame is not limited to this.

  Reference numeral 702 denotes a terminal for inputting a display control signal related to decoding output from the display control unit 603 in FIG. As the display control signal, the layer information to be decoded and the position information of the tile to be decoded are input. Further, the display control signal input to terminal 702 is input to separation section 704, base layer decoding section 707, and enhancement layer decoding section 710. A buffer 703 stores one frame of hierarchical code data input from the terminal 701.

  Reference numeral 704 denotes a separation unit. The separation unit 704 separates the header code data, base layer code data, and each enhancement layer code data from the hierarchical code data input from the buffer 703. Furthermore, the separation unit 704 divides the hierarchical code data (base layer code data and each enhancement layer code data) separated for each layer into code data for each tile and outputs the divided code data. Each separated code data is output to header decoding section 705, base layer decoding section 707, and enhancement layer decoding section 710. Further, when outputting the code data separated for each tile to each processing unit, the separation unit 704 outputs the number of the tile to be output (decoding target tile) to the independent tile determination unit 706 as tile position information.

  Reference numeral 705 denotes a header decoding unit. The header decoding unit 705 decodes header code data in sequence units and picture units, and reproduces parameters necessary for decoding. In particular, when the MCTS SEI code exists in the header code data, the header decoding unit 705 also decodes this. In particular, the header decoding unit 705 decodes and reproduces the independent tile flag and the independent tile position information. Reference numeral 706 denotes an independent tile determination unit that determines whether a decoding target tile (decoding target tile) is an independent tile. The independent tile determination unit 706 determines whether the decoding target tile is an independent tile based on the independent tile flag and the independent tile position information input from the header decoding unit 705 and the decoding target tile position information input from the separation unit 704. Determine whether or not. Furthermore, the independent tile determination unit 706 inputs the determination result to the base layer decoding unit 707 and the enhancement layer decoding unit 710.

  Reference numeral 707 denotes a base layer decoding unit. The base layer decoding unit 707 decodes the code data of the tiles of the base layer separated by the separation unit 704, and generates a decoded image of the base layer. A frame memory 708 holds a decoded image of each tile of the base layer generated by the base layer decoding unit 707. Reference numeral 709 denotes an enlargement unit that enlarges the decoded image of the base layer to the resolution of the enhancement layer and generates an enlarged image. A selector 720 selects a desired decoded image from the decoded image of the base layer or the decoded image of the enhancement layer, and outputs the selected decoded image to the terminal 712. Reference numeral 712 denotes a terminal that outputs the decoded image input from the selector 720 to the outside of the image decoding unit 605.

  Reference numeral 710 denotes an enhancement layer decoding unit. The enhancement layer decoding unit 710 decodes the code data of the enhancement layer tile separated by the separation unit 704, and generates an enhancement layer decoded image. Reference numeral 711 denotes a frame memory which holds a decoded image of each tile of the enhancement layer generated by the enhancement layer decoding unit 710.

  The overall control unit 714 controls each processing unit of the image decoding unit 605 and transmits parameters between the processing units. In FIG. 1, connection between the overall control unit 714 and each processing unit in the image decoding unit 605 is omitted. The overall control unit 714 can control each processing unit in the image decoding unit 605 and read / write parameters between the processing units through either a parameter signal line or a register bus. In the present embodiment, the overall control unit 714 in FIG. 1 is installed in the image decoding unit 605, but the present invention is not limited to this. That is, the overall control unit 714 is installed outside the image decoding unit 605, and controls each processing unit in the image decoding unit 605 and reads / writes parameters between the processing units using either a parameter signal line or a register bus. You may go through.

  The image decoding operation in the image decoding unit 605 described above will be described below with reference to the flowchart shown in FIG.

  First, the case where the decoding target layer (decoding target layer) is only the base layer will be described. Here, it is assumed that the user instructs the display control unit 603 to decode and display the base layer in the bitstream input from the interface 601.

  In step S <b> 801, the header code data that is input from the terminal 701 and exists at the head of the bitstream is input to the header decoding unit 705 through processing by the buffer 703 and the separation unit 704. The header decoding unit 705 decodes video_parameter_set, which is one of the sequence headers. This video_parameter_set includes a vps_max_layers_minus1 code that represents the number of layers of layer encoding. In the present embodiment, the vps_max_layers_minus1 code is 1. Subsequently, the header decoding unit 705 decodes the Sequence parameter set code. The sequence parameter set code also includes vui_parameters. The vui_parameters includes a tile_boundaries_aligned_flag code that is tile position match information. In the present embodiment, the tile_boundaries_aligned_flag code is 1.

  In step S802, the header decoding unit 705 decodes the Picture parameter set code. Since decoding of these header code data is described in detail in Non-Patent Document 1, description thereof is omitted here.

  In step S803, the independent tile determination unit 706 determines whether there is an independent tile in the decoding target frame. Then, the determination result is an independent tile flag. Actually, it is determined whether or not MCTS SEI is present. If MCTS SEI exists in the header code data, the independent tile flag is set to 1, and the process proceeds to step S804. If MCTS SEI does not exist in the header code data, the independent tile availability flag is set to 0, and the process proceeds to step S805. In this embodiment, it is determined that MCTS SEI is present in the header code data, the independent tile flag is set to 1, and the process proceeds to step S804. When an independent tile exists in the decoding target frame, the tile_boundaries_aligned_flag code of vui_parameters that is tile position matching information needs to be 1. If the tile_boundaries_aligned_flag code of vui_parameters is not 1, the header decoding unit 705 may return an error and stop decoding. Further, the header decoding unit 705 inputs the independent tile flag to the independent tile determination unit 706, the base layer decoding unit 707, and the enhancement layer decoding unit 710.

  In step S804, the header decoding unit 705 decodes the MCTS SEI code, and acquires the independent tile flag and the independent tile position information.

  In step S <b> 805, the separation unit 704 inputs tile position information regarding the display portion input from the terminal 702. In this embodiment, display of the entire base layer is instructed. For this reason, the tiles related to the display portion are all the tiles of the base layer. That is, the separation unit 704 extracts the code data of the decoding target tile of the base layer from the buffer 703 in order of the tile number from the tile 0 and outputs the code data to the base layer decoding unit 707.

  In step S806, the independent tile determination unit 706 inputs the number of the decoding target tile from the separation unit 704. The independent tile determination unit 706 also receives the independent tile position information from the header decoding unit 705. In this embodiment, there is one independent tile set, and the independent tile position information is 5 and 6. The independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. If the tile number of the decoding target tile matches the tile number of the independent tile position information (YES in step S806), the independent tile determination unit 706 determines that the decoding target tile is an independent tile, and proceeds to step S807. If the tile number of the decoding target tile does not match the tile number of the independent tile position information (NO in step S806), it is determined that the decoding target tile is not an independent tile set tile, and the process proceeds to step S808.

  In step S807, the decoding target tile is an independent tile in the decoding target frame of the base layer. For this reason, the base layer decoding unit 707, in other frames of the decoded base layer, the independent tile in the independent tile set at a position relatively equal to the position of the decoding target tile, and the decoded in the decoding target tile Decoding is performed with reference to only pixels. That is, the base layer decoding unit 707 performs inter-frame prediction with reference to the decoded image of the independent tile in the independent tile set that is stored in the frame memory 708 and is relatively equal to the position of the decoding target tile. Further, the base layer decoding unit 707 performs intra prediction with reference to the decoded image in the decoding target tile stored in the frame memory 708. Then, the base layer decoding unit 707 stores the decoded image of the decoded base layer decoding target tile in the frame memory 708. The decoded image is referred to when decoding the tile thereafter. Also, the base layer decoding unit 707 outputs the decoded image of the tile of the base layer to the display unit 606 in FIG. 6 via the selector 720 and the terminal 712.

  In step S808, the decoding target tile is not an independent tile in the decoding target frame of the base layer. For this reason, the base layer decoding unit 707 performs decoding with reference to the decoded image of the base layer of the decoded frame and the decoded pixels of the base layer of the decoding target frame. That is, the base layer decoding unit 707 performs inter-frame prediction with reference to the decoded image stored in the frame memory 708. Furthermore, the base layer decoding unit 707 performs intra prediction with reference to the decoded image that has been decoded in the decoding target tile. Then, the base layer decoding unit 707 stores the decoded image of the decoded base layer decoding target tile in the frame memory 708. The decoded image is referred to when decoding the subsequent tiles. Also, the base layer decoding unit 707 outputs the decoded image of the tile of the base layer to the display unit 606 in FIG. 6 via the selector 720 and the terminal 712.

  In step S809, the overall control unit 714 determines whether the code data of all tiles for one frame of the base layer have been decoded. If it is determined that the decoding process of the encoded data of all the tiles for one frame of the base layer has not been completed (NO in step S809), the process returns to step S805, and the separation unit 704 extracts and outputs the next tile. ,continue processing. On the other hand, if it is determined that the decoding process of the code data of all tiles for one frame of the base layer has been completed (YES in step S809), the process proceeds to step S810.

  In step S810, the separation unit 704 determines whether or not the layer to be decoded and displayed includes an enhancement layer based on the display control signal input from the display control unit 603 in FIG. 6 via the terminal 702. To do. If decoding and display of the enhancement layer is instructed (YES in step S810), the process proceeds to step S811, and if not (NO in step S810), the process proceeds to step S818. Here, since it is decoding of only a base layer, it progresses to step S818 and the enhancement layer decoding part 710 does not perform a decoding process.

  In step S818, overall control unit 714 determines whether or not the decoding process of the base layer code data or the enhancement layer code data of all the frames included in the sequence input from terminal 701 has been completed. Here, the overall control unit 714 determines whether or not the decoding of the code data of the base layer of all frames has been completed. If there is base layer or enhancement layer code data that has not been decoded (NO in step S818), the process advances to step S805 to process the next frame. If there is no code data for a frame that has not been decoded (YES in step S818), the decoding process ends.

  Note that the image decoded by the image decoding unit 605 is output to the display unit 606 of FIG. The display unit 606 displays the entire base layer decoded image output from the image decoding unit 605 when the display control unit 603 instructs the display of the base layer image.

  When the display control unit 603 instructs the display of the base layer of the recorded video according to the user's instruction, the input of the selector 604 is the storage unit 602. Then, the display control unit 603 performs control to select a necessary bit stream from the storage unit 602 and output it to the selector 604.

  Next, a case where the decoding target layer is an enhancement layer will be described. Decoding processing when the user instructs the display control unit 603 to decode and partially display the enhancement layer of the bitstream input from the interface 601 will be described. This corresponds to a case where a part of an image taken by a monitoring camera or the like is monitored in detail. The image decoding unit 605 is instructed by the display control unit 603 to specify the tile numbers included in the decoding and display areas of the base layer and the enhancement layer. In this embodiment, in order to simplify the description, the tiles included in the display area are the areas of the tile 5 and the tile 6 in FIG. Hereinafter, the decoding operation of the enhancement layer image in the image decoding unit 605 will be described based on the flowchart shown in FIG. 8 in the same manner as when the decoding and display of only the base layer is instructed. Also, the description of the part that performs the same operation as the decoding of only the base layer is simplified.

  In step S801, as instructed to display only the base layer, the header decoding unit 705 decodes the video_parameter_set and the Sequence parameter set. The header decoding unit 705 decodes the vps_max_layers_minus1 code and the tile_boundaries_aligned_flag code among them.

  In step S802, the header decoding unit 705 decodes the picture parameter set code in the same manner as when only the base layer is displayed.

  In step S803, as in the case of displaying only the base layer, the header decoding unit 705 determines whether there is an independent tile in the header code data.

  In step S804, the header decoding unit 705 decodes the MCTS SEI code and acquires the independent tile flag and the independent tile position information in the same manner as when only the base layer is displayed.

  In step S <b> 805, the separation unit 704 inputs tile position information regarding the display portion input from the terminal 702. In this description, the positions of the tiles that are instructed to be displayed are the tile 5 and the tile 6. Here, first, the separating unit 704 sets the decoding target tile as the tile 5 based on the position information of the tile instructed to be input, which is input from the terminal 702, and extracts the code data of the base layer of the tile 5, The extracted code data is output to base layer decoding section 707. Further, tile position information instructed to be displayed is input to the independent tile determination unit 706.

  In step S806, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, since tile 5 that is the decoding target tile is an independent tile, the process proceeds to step 807.

  In step S807, the decoding target tile is an independent tile. As in the case of displaying only the base layer, the base layer decoding unit 707 generates a decoded image by decoding the code data of the tile 5 of the base layer, and stores the decoded image in the frame memory 708. Since the enhancement layer is displayed here, the base layer decoding unit 707 does not output the generated decoded image from the terminal 712. However, the present invention is not limited to this, and the base layer decoding unit 707 can output a decoded image. In this case, both the decoded image generated by the base layer decoding unit 707 and the decoded image generated by the enhancement layer decoding unit 710 can be output and selected and displayed by the display unit 606.

  In step S809, the overall control unit 714 determines whether the code data of all tiles of the base layer related to the display portion input from the separation unit 704 has been decoded. Here, since the decoding of the code data of the tile 6 has not been completed, the process returns to step S806, and the code data of the base layer of the tile 6 is decoded.

  Hereinafter, decoding of the code data of the base layer of the tile 6 will be described.

  In step S805, the separation unit 704 extracts the code data of the base layer of the tile 6. In step S806, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Since the tile 6 that is the decoding target tile is an independent tile, the process proceeds to step S807. In step S807, the base layer decoding unit 707 decodes the code data of the base layer of the tile 6 and stores the decoded image in the frame memory 708.

  In step S809, the overall control unit 714 determines that the code data of all the tiles of the base layer related to the display portion input from the separation unit 704 has been decoded, and proceeds to step S810.

  In step S810, the separation unit 704 determines whether or not an extension layer is included in the layer to be displayed, based on the display control signal input from the display control unit 603 in FIG. 6 via the terminal 702. Here, since the enhancement layer is displayed, the process advances to step S811.

  In step S811, as in step S805, the separation unit 704 inputs tile position information relating to the display portion input from the terminal 702. Here, the positions of the tiles that are instructed to be displayed are the tile 5 and the tile 6. The separation unit 704 extracts the code data of the enhancement layer of the tile 5 that is the decoding target tile based on the input position information of the tile that is instructed to be displayed, and the enhancement layer decoding unit 710 extracts the extracted code data. Output to. Further, tile position information instructed to be displayed is input to the independent tile determination unit 706.

  In step S812, as in step S806, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. If the tile numbers match, the process proceeds to step S813, and if not, the process proceeds to step S815. Here, the independent tile position information is 5 and 6. Therefore, the independent tile determination unit 706 determines that the tile 5 that is the decoding target tile is a tile of the independent tile set, and proceeds to step S813.

  In step S813, the decoding target tile is an independent tile in the decoding target frame of the enhancement layer. The enlargement unit 709 inputs a decoded image included in an independent tile set at a position that is relatively equal to the position of the decoding target tile from the decoded image of the base layer that is stored in the frame memory 708. The enlargement unit 709 uses only the input decoded image of the independent tile to generate an enlarged image by filtering or the like, and outputs the enlarged image to the enhancement layer decoding unit 710.

  In step S814, enhancement layer decoding section 710 decodes the enhancement layer code data of the decoding target tile input from separation section 704. The enhancement layer decoding unit 710 refers to the enlarged image input from the enlargement unit 709, the decoded enhancement layer decoded image stored in the frame memory 711, and the decoded pixel of the decoding target tile. Is generated. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the base layer generated in step S813. Also, the enhancement layer decoding unit 710 refers to the decoded image in the independent tile set at a position relatively equal to the position of the decoding target tile among the decoded images of the enhancement layer stored in the frame memory 711, and performs inter-frame prediction. I do. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. Specifically, referring to FIG. 2, when decoding the tile 5 of the frame 206, the enlarged image of the frame 204, the decoded images of the tiles 5 and 6 of the decoded frame 205, and the tile 5 of the frame 206. Decoding is performed with reference to the decoded pixels. The decoded image of the enhancement layer tile generated by the enhancement layer decoding unit 710 is output to the frame memory 711 and held in the frame memory 711. Also, the decoded image of the enhancement layer generated by the enhancement layer decoding unit 710 is output to the display unit 606 in FIG. 6 via the selector 720 and the terminal 712.

  In step S817, overall control unit 714 determines whether or not the code data of all tiles of the enhancement layer relating to the display portion input from separation unit 704 has been decoded. Here, since the decoding of the code data of the enhancement layer of tile 6 has not been completed, the process returns to step S811, and the code data of the enhancement layer of tile 6 is decoded.

  Hereinafter, decoding of the code data of the enhancement layer of tile 6 will be described.

  In step S811, the code data of the enhancement layer of tile 6 is extracted. In step S812, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, since the tile 6 that is the decoding target tile is an independent tile, the process advances to step S813.

  In step S813, the enlargement unit 709 generates an enlarged image using only the input decoded image of the independent tile.

  In step S814, enhancement layer decoding section 710 decodes the enhancement layer code data of tile 6 to generate a decoded image, and stores the decoded image in frame memory 711. The enhancement layer decoding unit 710 performs decoding of the enhancement layer code data of the tile 6, the enlarged image input from the enlargement unit 709, the decoded enhancement layer decoded image stored in the frame memory 711, and the decoding target tile To the decoded pixel. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the base layer generated in step S813. Also, the enhancement layer decoding unit 710 performs inter-frame prediction with reference to the decoded image in the independent tile set at a position that is relatively equal to the position of the decoding target tile in the enhancement layer stored in the frame memory 711. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. Specifically, referring to FIG. 2, when decoding tile 6 of frame 206, an enlarged image of frame 204, decoded images of tile 5 and tile 6 of decoded frame 205, and tile 6 of frame 206. Decoding is performed with reference to the decoded pixels. The decoded image of the enhancement layer tile generated by the enhancement layer decoding unit 710 is output to the frame memory 711 and held in the frame memory 711. Also, the decoded image of the enhancement layer generated by the enhancement layer decoding unit 710 is output to the display unit 606 in FIG. 6 via the selector 720 and the terminal 712.

  In step S817, overall control unit 714 determines that the encoded data of all tiles of the enhancement layer related to the display portion has been decoded, and proceeds to step S818.

  In step S818, overall control unit 714 determines whether or not the decoding process of the encoded data of the tiles related to the display portion of all the frames included in the sequence input from terminal 701 has been completed. If there is a frame that has not been decoded (NO in step S818), the process advances to step S805 to process the next frame. If there is no frame that has not been decoded (YES in step S818), the decoding process ends.

  The case where the display area (decoding target tile) is configured with an independent tile set has been described above, but the case where it is not configured with an independent tile will be described. Steps up to step S805 are as described above.

  In step S806, the independent tile determination unit 706 determines that the decoding target tile is not an independent tile, and the process advances to step S808. In step S808, as in the case where the decoding target layer is only the base layer, the base layer decoding unit 707 generates a decoded image by decoding the tiles of the base layer, and stores the decoded image in the frame memory 708. Since the enhancement layer is displayed here, the base layer decoding unit 707 does not output the generated decoded image from the terminal 712.

  In step S809, the overall control unit 714 determines whether the code data of all tiles for one frame of the base layer have been decoded. Here, overall control unit 714 determines that the encoded data of all tiles for one frame of the base layer has been decoded, and proceeds to step S810. In step S810, the separation unit 704 determines that it is instructed to display up to the enhancement layer based on the input display control signal, and proceeds to step S811. In step S <b> 811, the separation unit 704 inputs tile position information related to the display portion from the terminal 702. Based on the input position information, the separation unit 704 extracts code data of the enhancement layer of the tile that is the decoding target tile. In step S812, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, the independent tile determination unit 706 determines that the decoding target tile is not a tile of the independent tile set (the tile number of the decoding target tile does not match the tile number of the independent tile position information), and the process proceeds to step S815.

  In step S815, the decoding target tile is not an independent tile. The enlarging unit 709 extracts, from the decoded base layer decoded image stored in the frame memory 708, the base layer tile at a position relatively equal to the position of the decoding target tile and the decoded images around the tile. input. The enlarging unit 709 uses the input decoded image of the base layer to generate an enlarged image by filtering or the like, and outputs the enlarged image to the enhancement layer decoding unit 710.

  In step S816, enhancement layer decoding section 710 decodes the enhancement layer code data of the decoding target tile input from separation section 704. The enhancement layer decoding unit 710 refers to the enlarged image input from the enlargement unit 709, the decoded enhancement layer decoded image stored in the frame memory 711, and the decoded pixel of the decoding target tile. Is generated. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the base layer generated in step S815. Further, the enhancement layer decoding unit 710 performs inter-frame prediction with reference to the enhancement layer decoded image stored in the frame memory 711. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. The decoded image of the enhancement layer tile generated by the enhancement layer decoding unit 710 is output to the frame memory 711 and held in the frame memory 711. Also, the decoded image of the enhancement layer generated by the enhancement layer decoding unit 710 is output to the display unit 606 in FIG.

  In step S817, the overall control unit 714 determines whether or not all related tiles have been decoded based on tile position information regarding the display portion input from the terminal 702 to the separation unit 704. If all tiles have not been decoded (NO in step S817), the process returns to step S811, and the separation unit 704 extracts and outputs the next tile, and continues the process. If the decoding process of the code data of all tiles related to the display portion has been completed (YES in step S817), the process proceeds to step S818.

  In step S818, overall control unit 714 determines whether or not the decoding processing of the code data for all the frames has been completed. If there is code data that has not been decoded (NO in step S818), the process proceeds to step S805 to process the next frame. If there is no code data that has not been decoded (YES in step S818), the decoding process ends.

  Returning to FIG. 6, the display unit 606 is instructed by the display control unit 603 to display the enhancement layer image. Therefore, the display unit 606 displays the enhancement layer decoded image decoded by the image decoding unit 605. Since the enhancement layer has a higher resolution than the base layer, the display unit 606 can obtain the effect of enlarging and displaying a part of the base layer image by displaying the decoded image of the enhancement layer.

  When the display control unit 603 instructs the display of the base layer of the recorded video according to the user's instruction, the input of the selector 604 is the storage unit 602. Then, the display control unit 603 performs control to select a necessary bit stream from the storage unit 602 and output it to the selector 604.

  With the above configuration and operation, when using independent tiles and independent tile sets, the relative positions of the tiles of the enhancement layer and the base layer can be matched. In other words, if the tile is an independent decoding tile set in the base layer, the tiles at positions relatively equal to the positions of the tiles in all enhancement layers can be used as the tiles of the independent decoding tile set. As a result, when decoding a hierarchically encoded bitstream, independent tiles can be decoded only by referring to the minimum image data in any hierarchy. As described above, by reducing the image data referred to in the prediction, it is possible to suppress the data transfer amount, reduce the calculation amount, and realize low power consumption. Further, in independent tile decoding processing, high-speed processing can be performed by independently decoding each layer from the basic layer to the enhancement layer without referring to tiles other than the independent tile. In particular, when a bit stream is generated by encoding so that an independent tile set is applied to an important area on the encoding side, the important area can be decoded at high speed when the bit stream is decoded. .

  In the present embodiment, as shown in FIG. 2, an example is shown in which only a frame temporally prior to a decoding target frame is predicted and decoded as a reference frame, but the present invention is not limited to this. That is, it is clear from the above description that the same reference is made in the case of prediction and decoding with reference to a plurality of frames.

  Moreover, although the image decoding part 605 using the expansion part 709 was demonstrated in this embodiment, this invention is not limited to this. That is, the enlargement unit 709 may be omitted. Alternatively, the expansion rate may be set to 1, and the quantization parameter decoded by the enhancement layer decoding unit 710 may be made smaller than the quantization parameter decoded by the base layer decoding unit 707. This makes it possible to decode the SNR layer data.

  In the present embodiment, an example in which code data of all layers is included in one frame of code data has been described. However, the present invention is not limited to this and may be input for each layer. For example, the code data may be stored together in the storage unit 602 for each layer and the code data may be cut out and read out from the extension layer as necessary.

  Further, in the present embodiment, the case where there is a base layer and one extended layer (two layers in total) has been described, but the present invention is not limited to this, and there may be three or more layers in total. In this case, by providing the enhancement layer decoding unit 710, the frame memory 711, and the expansion unit 709 as one set and providing the set for the number of enhancement layer hierarchies, it is possible to deal with more hierarchies. Also, as shown in FIG. 9, one enhancement layer decoding unit 710, one frame memory 911, and one enlargement unit 909 may be provided and used in the decoding of each layer. FIG. 9 is a block diagram of an image decoding apparatus that can decode an enhancement layer of a plurality of hierarchies and includes an enhancement layer decoding unit 710, a frame memory 911, and an enlargement unit 909 one by one. In FIG. 9, the same functions as those of the respective processing units of the image decoding unit 605 in FIG. Reference numeral 908 denotes a frame memory, which holds the decoded image generated by the base layer decoding unit 707. The frame memory 908 is different from the frame memory 708 of FIG. 7 in that a function for outputting to the selector 920 is added. Reference numeral 909 denotes an enlargement unit, which is different from the enlargement unit 709 in FIG. 7 in that the input from the frame memory 911 and the input from the frame memory 908 are selected and input is possible. Reference numeral 911 denotes a frame memory, which is different from the frame memory 711 in FIG. 7 in that a function of outputting code data of a desired tile to the enlargement unit 909 and the selector 920 is given. A selector 920 selects and inputs a desired decoded image from the frame memory 908 or the frame memory 911, and outputs the selected decoded image to the terminal 912. Reference numeral 912 denotes a terminal which outputs the decoded image input from the selector 920 to the outside of the image decoding unit 605.

  When the decoding process is performed using the image decoding unit 605 illustrated in FIG. 9, the operation of each processing unit will be described below with reference to the flowchart illustrated in FIG. FIG. 10 shows only a part obtained by changing step S805 to step S818 in FIG. 10, the steps having the same functions as those in FIG. 8 are given the same numbers as those in FIG. Further, in the present embodiment, an example of decoding a bitstream having the number of layers of 3 that is the bitstream generated by the encoding method of FIG. 5 by the image encoding apparatus 400 illustrated in FIG. 4 of the first embodiment. Will be described. In steps S801 to S804 in FIG. 8, the header decoding unit 705 decodes the header code data as described above. Here, the vps_max_layers_minus1 code is 2.

  First, the case where the decoding target layer is only the base layer will be described. Here, it is assumed that the user instructs the display control unit 603 to start decoding and display of the entire base layer in the bitstream input from the interface 601. Hereinafter, it is assumed that the decoding of one frame of the base layer is completed in steps S805 to S809 in FIG. However, all the decoded images generated by the base layer decoding unit 707 are stored in the frame memory 908.

  In step S1010, base layer decoding section 707 or enhancement layer decoding section 710 compares the number of decoded hierarchies with the display hierarchy indicated by display control section 603, and whether the display hierarchy has been decoded. Determine whether or not. If the number of decoded layers has reached the displayed layer (YES in step S1010), the process proceeds to step S1003. If not reached (NO in step S1010), the process proceeds to step S1001. Here, it is assumed that the separation unit 704 determines that the layer to be displayed is only the base layer based on the display control signal input from the terminal 702. For this reason, the base layer decoding unit 707 determines that the hierarchy to be displayed in step S1010 has been reached, and proceeds to step S1003.

  In step S1003, the selector 920 selects the decoded image of the lowest hierarchy among the decoded hierarchies. In this case, since the lowest layer is the base layer, the selector 920 reads the decoded image of the base layer decoded from the frame memory 908, and the read decoded image is displayed on the display unit 606 in FIG. Output. The display unit 606 displays the entire decoded image of the base layer output from the image decoding unit 605 when the display control unit 603 instructs the display of the image of the base layer.

  Next, a case where the decoding target layer is an enhancement layer will be described. Here, a description will be given of decoding when the user instructs the display control unit 603 to decode the enhancement layer in the bitstream input from the interface 601 and display a part of the decoded image of the enhancement layer. In addition, as an example, the hierarchy to be displayed is described as the second enhancement layer (the number of hierarchies is 3). Further, in the present embodiment, for the sake of simplicity, the tiles included in the display area are the areas of the tile 5 and the tile 6 in FIG. The decoding operation will be described based on the flowchart shown in FIG. 10 in the same manner as when decoding and display of only the base layer is instructed. Also, the description of the part that performs the same operation as the decoding of only the base layer is simplified.

  In step S806, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, since tile 5 that is the decoding target tile is an independent tile, the process proceeds to step 807. In step S807, the base layer decoding unit 707 decodes the decoded data of the tile 5 of the base layer to generate a decoded image, and stores the decoded image in the frame memory 908. In step S809, the overall control unit 714 determines whether the code data of all tiles of the base layer related to the display portion input from the separation unit 704 has been decoded.

  In step S1010, base layer decoding section 707 or enhancement layer decoding section 710 compares the number of decoded hierarchies with the display hierarchy indicated by display control section 603, and whether the display hierarchy has been decoded. Determine whether or not. Here, according to the display control signal input from the terminal 702, the layer to be displayed is the second enhancement layer (the number of layers is 3). Therefore, enhancement layer decoding section 710 determines that the hierarchy to be displayed has not been decoded, and proceeds to step S1001.

  In step S1001, enhancement layer decoding section 710 sets the base layer decoded in steps S807 to S808 or the enhancement layer of the hierarchy decoded in steps S1014 to S1016 described later as an upper layer. Further, the subsequent enhancement target decoding layer is set as a lower layer. Initially, the base layer encoded in steps S807 to S808 is set as an upper layer, and the first enhancement layer is set as a lower layer.

  In step S <b> 1011, the separation unit 704 inputs tile position information regarding the display portion input from the terminal 702. In this description, the positions of the tiles related to the display portion are the tile 5 and the tile 6. Then, based on the position information input from the terminal 702, the separation unit 704 converts the code data of the lower layer (first enhancement layer) of the tile 5 that is the decoding target tile among the hierarchical code data stored in the buffer 703. Extract. Furthermore, the separation unit 704 outputs the extracted code data to the enhancement layer decoding unit 710. Further, the separation unit 704 inputs the tile position information to the independent tile determination unit 706.

  In step S812, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. If the tile numbers match, the process proceeds to step S1013. If the tile numbers do not match, the process proceeds to step S1015. Here, the independent tile position information is 5 and 6, and the tile 5 that is the decoding target tile matches the tile number of the independent tile position information. Therefore, the independent tile determination unit 706 determines that the decoding target tile is a tile of the independent tile set, and proceeds to step S1013.

  In step S1013, the decoding target tile is an independent tile. Since the upper layer is the base layer, the enlargement unit 909 determines the independent tiles included in the independent tile set at a position relatively equal to the position of the decoding target tile from the decoded image of the base layer stored in the frame memory 908. Input the decoded image. The enlargement unit 909 uses only the input decoded image of the independent tile to generate an enlarged image by filtering or the like, and outputs the enlarged image to the enhancement layer decoding unit 710.

  In step S1014, as in step S814, the enhancement layer decoding unit 710 decodes the code data of the lower layer (first enhancement layer) of the decoding target tile input from the separation unit 704. The enhancement layer decoding unit 710, the enlarged image input from the enlargement unit 909, the decoded image of the decoded enhancement layer (first enhancement layer) stored in the frame memory 911, and the decoded pixels of the decoding target tile To generate a decoded image. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the upper layer (base layer) generated in step S1013. Also, the enhancement layer decoding unit 710 displays the decoded image in the independent tile set at a position relatively equal to the position of the decoding target tile among the decoded images of the lower layer (first enhancement layer) stored in the frame memory 911. Inter-frame prediction is performed with reference. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. The decoded image of the lower layer (first enhancement layer) tile decoded by the enhancement layer decoding unit 710 is output to the frame memory 911 and held in the frame memory 911.

  In step S1017, overall control unit 714 determines whether or not the code data of all tiles in the lower layer (first enhancement layer) related to the display portion input from separation unit 704 has been decoded. Here, since the decoding of the code data of the enhancement layer of tile 6 has not been completed, the process returns to step S1011 and the code data of the lower layer (first enhancement layer) of tile 6 is decoded.

  Hereinafter, decoding of the code data of the lower layer (first enhancement layer) of the tile 6 will be described.

  In step S1011, the separation unit 704 extracts code data of the lower layer (first enhancement layer) of the tile 6 that is the decoding target tile from the hierarchical code data stored in the buffer 703. In step S812, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, the independent tile determination unit 706 determines that the tile 6 that is the decoding target tile is an independent tile, and proceeds to step S1013.

  In step S1013, the enlargement unit 909 generates an enlarged image using only the input decoded image of the independent layer of the higher layer (basic layer). That is, the enlargement unit 909 receives the decoded image from the frame memory 908 and enlarges it by filtering or the like to generate an enlarged image.

  In step S1014, enhancement layer decoding section 710 decodes the code data of the lower layer (first enhancement layer) of tile 6 to generate a decoded image, and stores the decoded image in frame memory 911. The enhancement layer decoding unit 710 decodes the encoded data of the lower layer (first enhancement layer) of the tile 6, the enlarged image input from the enlargement unit 909, and the decoded image of the decoded enhancement layer stored in the frame memory 911 And the decoded pixel of the decoding target tile. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the upper layer (base layer) generated in step S1013. Also, the enhancement layer decoding unit 710 displays the decoded image in the independent tile set at a position relatively equal to the position of the decoding target tile among the decoded images of the lower layer (first enhancement layer) stored in the frame memory 911. Inter-frame prediction is performed with reference. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. Further, the decoded image of the lower layer (first enhancement layer) tile decoded by the enhancement layer decoding unit 710 is output to the frame memory 911 and held in the frame memory 911.

  In step S1017, overall control unit 714 determines that the encoded data of all tiles of the lower layer (first enhancement layer) related to the display portion has been decoded, and proceeds to step S1002.

  In step S1002, overall control unit 714 determines whether or not encoding has been completed for all layers represented by the decoded vps_max_layers_minus1 code. If decoding of tiles in all layers is not completed (NO in step S1002), the process returns to step S1010 to determine display. If decoding of tiles in all layers has been completed (YES in step S1002), the process proceeds to step S1003. Here, since the enhancement layer decoding process has not ended, the enhancement layer decoding unit 710 determines that the decoding process of tiles of all layers has not been completed, and returns to step S1010.

  Hereinafter, decoding of the second enhancement layer is performed. That is, in step S1010, enhancement layer decoding section 710 determines whether or not the hierarchy to be displayed has been decoded. According to the display control signal input from the terminal 702, the display layer is the second enhancement layer. Here, enhancement layer decoding section 710 proceeds to step S1001 in order to determine that only the first enhancement layer has been decoded (the second enhancement layer has not been decoded). In step S1001, enhancement layer decoding section 710 sets the first enhancement layer decoded in steps S1014 to S1016 as an upper layer and the second enhancement layer as a lower layer.

  In step S <b> 1011, the separation unit 704 extracts code data of a lower layer (second enhancement layer) tile from the hierarchical code data stored in the buffer 703, and inputs the code data to the enhancement layer decoding unit 710. Here, first, the separation unit 704 extracts code data of a lower layer (second enhancement layer) of the tile 5 and inputs the extracted code data to the enhancement layer decoding unit 710. In step S812, the independent tile determination unit 706 determines that the tile 5 that is the decoding target tile is an independent tile, and proceeds to step S1013. In step S1013, the enlargement unit 909 has the enhancement layer (first enhancement layer) as the upper layer. Therefore, the enlargement unit 909 decodes the independent tiles included in the independent tile set at a position relatively equal to the position of the decoding target tile from the decoded image of the upper layer (first enhancement layer) stored in the frame memory 908. Enter an image. That is, the enlargement unit 909 uses only the decoded image of the input independent tile of the upper layer (first enhancement layer) to generate an enlarged image by filtering or the like, and sends the enlarged image to the enhancement layer decoding unit 710. input.

  In step S1014, enhancement layer decoding section 710 decodes the code data of the lower layer (second enhancement layer) of the decoding target tile input from separation section 704. The enhancement layer decoding unit 710 generates a decoded image with reference to the next image. That is, the enhancement layer decoding unit 710 decodes the enlarged image of the upper layer (first enhancement layer hierarchy) input from the enlargement unit 909 and the decoded enhancement layer (second enhancement layer) stored in the frame memory 911. The image and the decoded pixel of the decoding target tile are referred to. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the upper layer (first enhancement layer) generated in step S1013. Also, the enhancement layer decoding unit 710 displays the decoded image in the independent tile set at a position relatively equal to the position of the decoding target tile among the decoded images of the lower layer (first enhancement layer) stored in the frame memory 911. Inter-frame prediction is performed with reference. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. Further, the decoded image of the lower layer (second enhancement layer) tile decoded by the enhancement layer decoding unit 710 is output to the frame memory 911 and held in the frame memory 911.

  In step S1017, overall control unit 714 determines whether or not the code data of all tiles in the lower layer (second enhancement layer) related to the display portion input from separation unit 704 has been decoded. Here, since the decoding of the code data of the enhancement layer of tile 6 has not been completed, the process returns to step S1011 and the code data of the lower layer (second enhancement layer) of tile 6 is decoded. The decoding of the lower layer of tile 6 is the same as the decoding process of the code data of the second enhancement layer of tile 5 as described above, assuming that the upper layer is the first enhancement layer hierarchy and the lower layer is the second enhancement layer. Since there is, description is abbreviate | omitted.

  In step S1002, since the overall control unit 714 has decoded up to the second enhancement layer, the overall control unit 714 determines that the decoding process for all the tiles has been completed, and the process advances to step S1003. In step S1003, the selector 920 selects the decoded image of the lowest hierarchy among the decoded hierarchies. In this case, since the lowest hierarchy is the second enhancement layer, the selector 920 reads out the decoded image of the second enhancement layer from the frame memory 911 and sends the decoded image of the second enhancement layer to the terminal 912 in FIG. The data is output to the display unit 606. The display unit 606 displays the entire decoded image of the second enhancement layer output from the image decoding unit 605 when the display control unit 603 instructs the display of the second enhancement layer image. .

  In the above description, the hierarchy to be displayed is described as the second enhancement layer (the number of hierarchies is 3). However, when the number of hierarchies of the encoded data of the hierarchical coding is 3 or more and the hierarchy to be displayed is the first enhancement layer (the number of hierarchies is 2), decoding of the first enhancement layer is completed (NO in step S1002). ) After that, the process proceeds to step S1003 in step S1010. For this reason, the decoding of the decoding of the code data of a hierarchy higher than a 2nd enhancement layer is not performed.

  The case where the display area (decoding target tile) is configured with an independent tile set has been described above, but the case where it is not configured with an independent tile will be described. Steps up to step S805 are as described above.

  In step S806, the independent tile determination unit 706 determines that the decoding target tile is not an independent tile, and the process advances to step S808. In step S808, as in the case where the decoding target layer is only the base layer, the base layer decoding unit 707 generates a decoded image by decoding the tiles of the base layer, and stores the decoded image in the frame memory 908.

  In step S809, the overall control unit 714 determines whether the code data of all tiles for one frame of the base layer have been decoded. Here, base layer decoding section 707 determines that the encoded data of all tiles for one frame of the base layer has been decoded, and proceeds to step S1010. In step S1010, since base layer decoding section 707 or enhancement layer decoding section 710 displays up to the second enhancement layer, it determines that the layer to be displayed has not been decoded, and proceeds to step S1001.

  In step S1001, enhancement layer decoding section 710 sets the base layer decoded in step S808 as an upper layer, and sets the subsequent decoding target enhancement layer (first enhancement layer) as a lower layer. In step S <b> 1011, the separation unit 704 inputs tile position information regarding the display portion input from the terminal 702. Then, based on the input position information, the separation unit 704 extracts code data of a lower layer (first enhancement layer) of the decoding target tile from the hierarchical code data stored in the buffer 703. In step S812, the independent tile determination unit 706 compares the tile number of the decoding target tile with the tile number of the independent tile position information. Here, the tile 5 that is the decoding target tile does not match the tile number of the independent tile position information. Therefore, the independent tile determination unit 706 determines that the decoding target tile is not a tile of the independent tile set, and the process proceeds to step S1015.

  In step S1015, the enlargement unit 909 determines, from the decoded image of the upper layer (basic layer) stored in the frame memory 908, the tile of the basic layer at a position relatively equal to the position of the decoding target tile, and the periphery of the tile The decoded image is input. The enlargement unit 909 uses only the input decoded image of the base layer tile to generate an enlarged image by filtering or the like, and outputs the enlarged image to the enhancement layer decoding unit 710.

  In step S1016, enhancement layer decoding section 710 decodes the code data of the lower layer (first enhancement layer) of the decoding target tile input from separation section 704. The enhancement layer decoding unit 710 generates a prediction image with reference to the following. That is, the enlarged image of the upper layer (basic layer) input from the enlargement unit 909, the decoded image of the decoded lower layer (first enhancement layer) stored in the frame memory 911, and the lower layer ( Reference is made to the decoded pixels of the first enhancement layer. Furthermore, the enhancement layer decoding unit 710 generates a decoded image from the predicted image generated by reference and the decoded prediction error. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the upper layer (base layer) generated in step S1015. Further, the enhancement layer decoding unit 710 performs inter-frame prediction with reference to the decoded image of the lower layer (first enhancement layer) stored in the frame memory 711. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile of the lower layer (first enhancement layer). The decoded image of the lower layer (first enhancement layer) tile generated by the enhancement layer decoding unit 710 is output to the frame memory 911 and held in the frame memory 911.

  In step S1017, overall control unit 714 determines whether or not the decoding processing of the code data of all tiles of the lower layer (first enhancement layer) related to the display portion has been completed. Here, the enhancement layer decoding unit 710 determines that the decoding processing of the code data of all the tiles of the first enhancement layer has been completed, and proceeds to step S1002. In step S1002, overall control unit 714 determines whether or not the decoding process has been completed for all layers. Here, enhancement layer decoding section 710 determines that the decoding process of the second enhancement layer has not ended, and returns to step S1010.

  Hereinafter, decoding of the second enhancement layer is performed. In step S1010, enhancement layer decoding section 710 determines that decoding of the second enhancement layer hierarchy that is the hierarchy to be displayed has not been completed, and proceeds to step S1001. In step S1001, enhancement layer decoding section 710 sets the first enhancement layer decoded in step S1016 as the upper layer and the second enhancement layer as the lower layer. In step S1011, the separation unit 704 extracts code data of the decoding target tile of the lower layer (second enhancement layer), and outputs the code data to the enhancement layer decoding unit 710. In step S812, the independent tile determination unit 706 determines that the decoding target tile is not a tile of the independent tile set, and proceeds to step S1015. In step S1015, since the upper layer is the enhancement layer (first enhancement layer), the enlargement unit 909 inputs the decoded image of the enhancement layer (first enhancement layer) layer stored in the frame memory 908. Then, the enlarging unit 909 generates an enlarged image by enlarging by filtering or the like using the input decoded image of the upper layer (first enhancement layer). At this time, an enlarged image may be generated using a tile at a position relatively equal to the position of the decoding target tile and pixels around the tile. Furthermore, the enlargement unit 909 inputs the generated enlarged image to the enhancement layer decoding unit 710.

  In step S1014, enhancement layer decoding section 710 decodes the code data of the lower layer (second enhancement layer) of the decoding target tile input from separation section 704. The enhancement layer decoding unit 710 receives the enlarged image of the upper layer (first enhancement layer) input from the enlargement unit 909, the decoded image of the decoded enhancement layer (second enhancement layer) stored in the frame memory 911, and the decoding A decoded image is generated with reference to the decoded pixels of the target tile. That is, the enhancement layer decoding unit 710 performs inter-layer prediction with reference to the enlarged image of the upper layer (first enhancement layer) generated in step S1015. Further, the enhancement layer decoding unit 710 performs inter-frame prediction with reference to the decoded image of the lower layer (first enhancement layer) stored in the frame memory 911. Furthermore, the enhancement layer decoding unit 710 performs intra prediction with reference to the decoded image in the decoding target tile. The decoded image of the lower layer (second enhancement layer) tile decoded by the enhancement layer decoding unit 710 is output to the frame memory 911 and held in the frame memory 911.

  In step S1017, overall control unit 714 determines whether or not the code data of all tiles in the lower layer (second enhancement layer) related to the display portion input from separation unit 704 has been decoded. Here, enhancement layer decoding section 710 determines that decoding of the code data of all tiles of the second enhancement layer has ended, and proceeds to step S1002. In step S1002, since the overall control unit 714 has decoded up to the second enhancement layer, the overall control unit 714 determines that the decoding process for all the tiles has been completed, and the process advances to step S1003. In step S1003, the selector 920 selects the decoded image of the lowest hierarchy among the decoded hierarchies. In this case, since the lowest hierarchy is the second enhancement layer, the selector 920 reads out the decoded image from the frame memory 911 and outputs the read out decoded image to the display unit 606 in FIG. The display unit 606 displays the second enhancement layer decoded image output from the image decoding unit 605 in response to an instruction to display the second enhancement layer image from the display control unit 603.

  In the above description, the hierarchy to be displayed is described as the second enhancement layer (the number of hierarchies is 3). However, when the number of hierarchies of the encoded data of the hierarchical coding is 3 or more and the hierarchy to be displayed is the first enhancement layer (the number of hierarchies is 2), decoding of the first enhancement layer is completed (NO in step S1002). ) After that, the process proceeds to step S1003 in step S1010. For this reason, the decoding of the code data of a hierarchy higher than a 2nd enhancement layer is not performed.

  With the above configuration and operation, the relative positions of the independent decoding tiles can be matched in each of the enhancement layer and the base layer. That is, when an independent tile is set as a predetermined tile in the base layer, in each extension layer, a tile at a position relatively equal to the position of the independent tile of the base layer can be an independent tile. This makes it possible to limit pixels to be referred to for prediction and decoding of code data of independent tiles in any layer of hierarchical encoding. In particular, in FIG. 6, if the tile instructed to be displayed by the display control unit 603 is an independent tile, necessary code data is read from the storage unit 602, and the image decoding unit 605 only has to decode the code data. For this reason, it becomes possible to process at higher speed than before.

  Further, when the MCTS SEI code is present in the bitstream, the tile_boundaries_aligned_flag code of vui_parameters which is tile position matching information is always set to 1. That is, when the MCTS SEI code is present in the bitstream in vui_parameters, the tile_boundaries_aligned_flag code as code data can be omitted. If the MCTS SEI code is not present in the bitstream, the tile_boundaries_aligned_flag code is decoded and referenced in the subsequent decoding. If the MCTS SEI code is in the bitstream, the tile_boundaries_aligned_flag code is not encoded, so the value of 1 is always set on the decoding side. By doing in this way, it becomes possible to perform the decoding similarly even if there is no tile_boundaries_aligned_flag code.

<Embodiment 3>
In the first embodiment and the second embodiment, each processing unit illustrated in FIGS. 1, 4, 6, 7, and 9 has been described as being configured by hardware. However, the processing performed in each processing unit shown in these figures may be executed by a computer program.

  FIG. 11 is a block diagram illustrating a hardware configuration example of a computer that executes processing performed by each processing unit of the image encoding device and the image decoding device according to the first and second embodiments.

  The CPU 1101 controls the entire computer using computer programs and data stored in the RAM 1102 and the ROM 1103, and performs the above-described processes performed by the image encoding device and the image decoding device according to each of the above-described embodiments. Execute. That is, the CPU 1101 functions as each processing unit shown in FIGS. 1, 4, 6, 7, and 9.

  The RAM 1102 has an area for temporarily storing computer programs and data loaded from the external storage device 1106, data acquired from the outside via an I / F (interface) 1107, and the like. Further, the RAM 1102 has a work area used when the CPU 1101 executes various processes. That is, the RAM 1102 can be allocated as, for example, a frame memory or can provide other various areas as appropriate.

  The ROM 1103 stores setting data of the computer, a boot program, and the like. The operation unit 1104 includes a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1101 by the user operating the computer. The output unit 1105 performs control for displaying the processing result by the CPU 1101. Further, the output unit 1105 performs control for displaying a processing result by the CPU 1601 in a display unit (not shown) configured by, for example, a liquid crystal display.

  The external storage device 1106 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1106 stores an operating system (OS) and computer programs for causing the CPU 1101 to realize the functions of the units illustrated in FIGS. 1, 4, 6, 7, and 9. Furthermore, each image data as a processing target may be stored in the external storage device 1106.

  Computer programs and data stored in the external storage device 1106 are appropriately loaded into the RAM 1102 under the control of the CPU 1101 and are processed by the CPU 1101. The I / F 1107 can be connected to other devices such as a network such as a LAN or the Internet, a projection device, a display device, etc., and the computer acquires and sends various information via the I / F 1107. Can be. A bus 1108 connects the above-described units.

  The operation in the above-described configuration is controlled by the CPU 1101 centering on the operation described in the above flowchart.

<Other embodiments>
In order to easily realize the present invention, it is useful to clearly indicate the presence or absence of an independent tile at a level close to the head of the bit stream. For example, a method using vui_parameters will be described with reference to FIG. FIG. 12 is a diagram illustrating the syntax of vui_parameters. In vui_parameters, a motion_constrained_tile_sets_flag code indicating that an independent tile always exists in the bitstream is included. If the value of this code is 1, it indicates that MCTS SEI is included, an independent tile exists in the bitstream, and the relative positions of the tiles of the base layer and the enhancement layer match. That is, since the tile_boundaries_aligned_flag code is always fixed at 1, there is no need to encode the tile_boundaries_aligned_flag code. On the other hand, if the tile_boundaries_aligned_flag code is 0, the MCTS SEI is not included and no independent tile exists in the bitstream. Therefore, it is necessary to encode the tile_boundaries_aligned_flag code. An image decoding apparatus that decodes such a bitstream can acquire information that an independent tile is included in the bitstream before performing the decoding process of each tile. For this reason, when decoding a specific area, the image decoding apparatus can perform high-speed decoding processing using independent tiles. Further, as a result, it is possible to determine whether an application such as partial enlarged display is valid before performing the decoding process of each tile.

  The size of the image, the number of tile divisions, and the position of the independent tile in one frame are not limited to the above-described embodiments.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (28)

An image encoding device that hierarchically encodes images constituting a moving image in a plurality of layers,
Generating means for generating a second image having a different hierarchy from the first image;
Encoding means for encoding a first region in the first image and a second region corresponding to the first region in the second image;
Information encoding means, and
The first region is a region where it is limited to refer to other than the first region in the first image, and the first region is at least a part of the second image. If it is a region encoded by reference,
The encoding means includes
Encoding the first region with reference to only the second region in the region of the second image;
Encoding the second region without referring to other than the second region of the second image;
The information encoding means encodes information indicating that reference to the outside of the second area is restricted when referring to the second image in the first area. Image encoding device. The image coding apparatus according to claim 1, wherein the first area and the second area are tiles. The image coding apparatus according to claim 2, wherein the first area and the second area are tiles that can be processed independently. SEI generation means for generating SEI including information indicating that the first area exists in the first image when the first area is set in the first image. The image coding device according to claim 1, wherein the image coding device is a video encoding device. The image encoding apparatus according to claim 4, wherein the SEI generation unit generates the SEI in units of sequences. When the first area exists in the first image, information on the first area and a code generated by encoding the first image and the second image by the encoding unit The image encoding apparatus according to claim 1, further comprising: an output unit that outputs the encoded data. The image encoding apparatus according to claim 1, wherein the generation unit generates the second image by reducing the first image. Local decoding means for reproducing a decoded image using encoded data generated by encoding an image by the encoding means;
The image encoding apparatus according to claim 1, further comprising an enlarging unit that enlarges the decoded image. The image coding apparatus according to claim 1, wherein the first image and the second image have different resolutions or image quality. The image coding apparatus according to any one of claims 1 to 9, wherein the second image is an image having a lower resolution or lower image quality than the first image. The first image is an enhancement layer;
The image coding apparatus according to any one of claims 1 to 10, wherein the second image is a base layer. 12. The SEI generation means for generating SEI including information on the position of the first area when the first area is set in the first image. The image encoding device according to claim 1. The second image is identified when the first region is identified as being present in the first image based on information regarding whether the first region is present in the first image. Setting means for setting the second region;
When an independent area that is an area to be encoded without referring to other areas in the encoding target image exists in the encoding target image, the independent area is included in the encoding target image. A header generation means for generating SEI based on at least one of a flag indicating that exists and information on a position of the independent region in the encoding target image;
The image according to any one of claims 1 to 12, wherein the first region in the first image and the second region in the second image exist at the same position. Encoding device. An image decoding device that decodes encoded data generated by hierarchically encoding images constituting a moving image in a plurality of layers,
Decoding means for decoding a first area in the first image and a second area corresponding to the first area in the second image having a different hierarchy from the first image;
Information decoding means for decoding information indicating that referring to the outside of the second area is restricted when referring to the second image in the first area, and
The first region is a region where it is limited to refer to other than the first region in the first image, and the first region is at least a part of the second image. If it is a region that is decoded by reference,
The decoding means includes
According to the information, the first region is decoded in the region of the second image with reference to only the second region,
The image decoding apparatus, wherein the second area is decoded without referring to the area other than the second area of the second image. The image decoding apparatus according to claim 14, wherein the first area and the second area are tiles. The image decoding apparatus according to claim 15, wherein the first area and the second area are tiles that can be processed independently. The derivation means for deriving information indicating that the first region exists in the first image from SEI of the encoded data, 17. Image decoding apparatus. The image decoding apparatus according to claim 17, wherein the SEI is assigned to the encoded data in units of sequences. The image decoding apparatus according to any one of claims 14 to 18, wherein the second image is an image generated by reducing the first image. The image decoding device according to any one of claims 14 to 19, wherein the first image and the second image are different in resolution or image quality. The image decoding device according to any one of claims 14 to 20, wherein the second image is an image having a lower resolution or lower image quality than the first image. The first image is an enhancement layer;
The image decoding apparatus according to any one of claims 14 to 21, wherein the second image is a base layer. The image decoding apparatus according to any one of claims 14 to 22, further comprising: a derivation unit that derives information on the position of the first region from the SEI of the encoded data. A flag indicating whether or not an independent region that is a region to be decoded without referring to another region in the decoding target image exists in the decoding target image, and the independent region in the decoding target image Deriving means for deriving at least one of the position information from the SEI of the encoded data;
The decoding means, when it is specified that the first region exists in the first image based on at least one of the flag and the information derived by the derivation means, Of the region without reference to other regions in the second image,
The image according to any one of claims 14 to 23, wherein the first region in the first image and the second region in the second image are present at the same position. Decoding device. An image encoding method for hierarchically encoding images constituting a moving image in a plurality of layers,
A generation step of generating a second image having a hierarchy different from that of the first image;
An encoding step of encoding a first region in the first image and a second region corresponding to the first region in the second image;
An information encoding process, and
The first region is a region where it is limited to refer to other than the first region in the first image, and the first region is at least a part of the second image. If it is a region encoded by reference,
In the encoding step,
Encoding the first region with reference to only the second region in the region of the second image;
Encoding the second region without referring to other than the second region of the second image;
An image encoding method comprising: encoding information indicating that referencing outside the second area is restricted when referring to the second image in the first area. An image decoding method for decoding encoded data generated by hierarchically encoding images constituting a moving image in a plurality of layers,
A decoding step of decoding a first area in the first image and a second area corresponding to the first area in the second image having a different hierarchy from the first image;
An information decoding step of decoding information indicating that referring to the outside of the second area is restricted when referring to the second image in the first area, and
The first region is a region where it is limited to refer to other than the first region in the first image, and the first region is at least a part of the second image. If it is a region that is decoded by reference,
In the decoding step,
According to the information, the first region is decoded in the region of the second image with reference to only the second region,
The image decoding method, wherein the second area is decoded without referring to areas other than the second area of the second image.   A program that causes a computer to function as each unit of the image encoding device according to any one of claims 1 to 13.   25. A program that causes a computer to function as each unit of the image decoding device according to claim 14.

Images