JP2018056686A - Image encoder, image encoding method and image encoding program, and image decoder, image decoding method and image decoding program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image encoder which enables the increase in encoding efficiency when an image produced by projecting a 360-degree whole sky image onto a hexahedron is encoded in such a way that the image is allocated to regions in a frame, except some regions.SOLUTION: In an image encoder, a picture division part 104 divides each frame of image information into one or more of first tiles set on an allocation region to which an image is allocated, and one or more of second tiles set on a non-allocation region; a coded data generation part encodes an image of the one or more of first tiles to generate coded data; a syntax generation part (entropy encoding part 5) generates syntax including syntax elements which define each of first and second tiles in frame by horizontal and vertical positions of an upper left corner portion, and horizontal and vertical sizes; and a bit stream generation part (entropy encoding part 5) generates bet streams including coded data and the syntax.SELECTED DRAWING: Figure 6

Description

  The present disclosure relates to an image encoding device, an image encoding method, an image encoding program, an image decoding device, an image decoding method, and an image decoding program.

  H.265 / MPEG-H HEVC (High Efficiency Video Coding) (hereinafter referred to as HEVC), which can encode image information more efficiently than H.264 / MPEG-4 AVC (Advanced Video Coding). Are abbreviated to be standardized) (see Non-Patent Document 1). In HEVC, tiles are defined that divide pixels constituting a frame (picture) into blocks called coding tree units (CTUs) as division units.

  In Non-Patent Document 2, as shown in FIG. 1, 360-degree all-sky image 401 visually recognized by a head-mounted display is projected onto surfaces 402A to 402F of hexahedron 402, and images projected onto surfaces 402A to 402F are illustrated in FIG. As shown in FIG. 2, it is described that the image is expanded into six regions in the frame. In FIG. 2, an area that is not hatched is an allocated area to which an image is assigned, and an area that is hatched is an unassigned area to which no image is assigned.

  A user wearing a head-mounted display can view the contents of a virtual reality image (VR image) spreading in a spherical shape in front of the eyes by viewing the image shown in FIG. 2 with the head-mounted display.

”High Efficiency Video Coding (HEVC) text specification draft 10 (for FDIS & Last Call)”, JCTVC-L1003, January 2013 “GoPro test sequences for Virtual Reality video coding”, JVET-C0021, May 2016

  However, in the HEVC tile, the image shown in FIG. 2 cannot be divided and encoded as shown in FIG. 3 or FIG. In the non-allocation area in FIG. 2, originally, it is unnecessary to transmit the encoded data to the image decoding apparatus and decode it. However, in HEVC, skip mode, direct mode, DC (direct current) mode in intra prediction, etc. Need to be encoded. Therefore, the image encoding device must transmit the code even in the non-allocation area, and the image decoding device must decode the transmitted code.

  The embodiment improves the encoding efficiency when encoding an image obtained by projecting a 360-degree all-sky image onto a hexahedron to an area excluding a part of the area in the frame, and the amount of encoding or decoding processing It is an object to provide an image encoding device, an image encoding method, an image encoding program, an image decoding device, an image decoding method, and an image decoding program.

  According to the first aspect of the embodiment, image information in a format including an allocated area in which an image is allocated in a frame and an unallocated area in which no image is allocated is input, and each frame of the image information is A picture dividing unit that divides into one or a plurality of first tiles set in the allocation area and one or a plurality of second tiles set in the non-allocation area, and an image of the first tile The encoded data generation unit that generates encoded data by encoding the image, and the first and second tiles are defined by the horizontal position and vertical position of the upper left corner in the frame, and the horizontal size and vertical size, respectively. Generating a syntax including the first syntax element, and generating a bitstream including the encoded data and the syntax The image coding apparatus is provided, characterized in that it comprises a appropriate bit stream generation unit.

  According to the second aspect of the embodiment, each frame of image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated is set as the allocation area. Dividing into one or a plurality of first tiles and one or a plurality of second tiles set in the non-allocation area, encoding an image of the first tiles to generate encoded data, Each of the first and second tiles generates a syntax including syntax elements defined by a horizontal position and a vertical position of the upper left corner in the frame, a horizontal size and a vertical size, and the encoded data; An image encoding method is provided that generates a bitstream including the syntax.

  According to the third aspect of the embodiment, each frame of image information in a format including an allocation area to which an image is allocated in a frame and a non-allocation area to which no image is allocated is assigned to the computer. Dividing into one or more first tiles set to 1 and a plurality of second tiles set to the non-allocation area, and encoding and encoding an image of the first tile Generating data, and generating a syntax including a syntax element for each of the first and second tiles defined by a horizontal position and a vertical position of the upper left corner in the frame, and a horizontal size and a vertical size. And generating a bitstream including the encoded data and the syntax. Image coding program is provided.

  According to the fourth aspect of the embodiment, it is image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and each frame of the image information includes: The image is divided into one or more first tiles set in the allocated area and one or more second tiles set in the non-allocated area, and the image of the first tile is encoded A syntax including encoded data and a first syntax element that defines each of the first and second tiles by a horizontal position and a vertical position of the upper left corner in the frame, and a horizontal size and a vertical size; A bitstream receiving unit that receives a bitstream including: a decoding unit that decodes the encoded data to generate a decoded image of the first tile; and the first An image comprising: a picture composition unit that composes a decoded image of the first tile and a single image of the second tile based on a syntax element to generate decoded image information of each frame. A decoding device is provided.

  According to the fifth aspect of the embodiment, the image information has a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and each frame of the image information includes: The image is divided into one or more first tiles set in the allocated area and one or more second tiles set in the non-allocated area, and the image of the first tile is encoded A bit including encoded data and a syntax including a syntax element in which each of the first and second tiles is defined by the horizontal position and vertical position of the upper left corner in the frame and the horizontal size and vertical size. Receiving a stream, decoding the encoded data to generate a decoded image of the first tile, and decoding the first tile based on the syntax element. Image decoding method characterized by by combining the single image of the image and the second tile to generate a decoded image information of each frame is provided.

  According to the sixth aspect of the embodiment, the image information is in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated in the computer. A frame is divided into one or a plurality of first tiles set in the allocated area and one or a plurality of second tiles set in the non-allocated area, and an image of the first tile is encoded. Encoded syntax data, and a syntax including syntax elements defined by the horizontal position and vertical position of the upper left corner of the frame, and the horizontal size and vertical size, respectively, of the first and second tiles. Receiving a bitstream including: decoding the encoded data to generate and receive a decoded image of the first tile; and And a step of generating a decoded image information of each frame by combining the decoded image of the first tile and the single image of the second tile based on a block element. A program is provided.

  According to the image encoding device, the image encoding method, and the image encoding program of the embodiment, and the image decoding device, the image decoding method, and the image decoding program, an image obtained by projecting a 360-degree all-sky image onto a hexahedron is obtained. It is possible to improve encoding efficiency when encoding is performed by allocating to a region excluding a part of the region in the frame, and the processing amount of encoding or decoding can be reduced.

FIG. 1 is a perspective view showing a 360 degree whole sky image and a hexahedron onto which the 360 degree whole sky image is projected. FIG. 2 is a diagram illustrating a state in which an image obtained by projecting a 360-degree all-sky image onto a hexahedron is developed into six regions in the frame. FIG. 3 is a diagram illustrating a first example that cannot be divided and encoded by HEVC tiles. FIG. 4 is a diagram illustrating a second example in which FIG. 3 cannot be divided and encoded by HEVC tiles. FIG. 5 is a block diagram illustrating an overall configuration example of an image encoding / decoding system including an image encoding device and an image decoding device. FIG. 6 is a block diagram illustrating an image encoding device according to an embodiment. FIG. 7 is a diagram illustrating a configuration example of a frame divided into a plurality of slices. FIG. 8 is a diagram illustrating a configuration example of a frame divided into a plurality of tiles. FIG. 9 is a diagram illustrating an example of the relationship between slices and tiles. FIG. 10 is a diagram showing the position and size of each tile. FIG. 11 is a diagram showing the position, size, and skipline flag of each tile in a table format. FIG. 12 is a diagram illustrating an example in which another tile is referred to when a certain tile is encoded or decoded. FIG. 13 is a diagram illustrating an example of syntax transmitted by the image encoding device according to the embodiment. FIG. 14 is a flowchart illustrating the operation of the image encoding device according to the embodiment, the processing by the image encoding method according to the embodiment, and the processing executed by the image encoding program according to the embodiment. FIG. 15A is a block diagram illustrating a schematic configuration of a computer including a storage unit that stores an image encoding program according to an embodiment. FIG. 15B is a block diagram illustrating a schematic configuration of a computer including a storage unit that stores an image decoding program according to an embodiment. FIG. 16 is a block diagram illustrating an image decoding apparatus according to an embodiment. FIG. 17 is a flowchart illustrating the operation of the image decoding apparatus according to the embodiment, the processing by the image decoding method according to the embodiment, and the processing executed by the image decoding program according to the embodiment. FIG. 18 is a diagram for explaining the first modification. FIG. 19 is a diagram for explaining the second modification. FIG. 20 is a diagram for explaining a third modification.

  Hereinafter, an image encoding device, an image encoding method, an image encoding program, an image decoding device, an image decoding method, and an image decoding program according to an embodiment will be described with reference to the accompanying drawings.

  First, an overall configuration example of an image encoding / decoding system including an image encoding device and an image decoding device will be described with reference to FIG. In FIG. 5, the preprocessing device 50 sets various conditions for encoding image information by the image encoding device 100 according to the embodiment in accordance with an operation by the user. The image encoding device 100 encodes the image information according to the setting information from the preprocessing device 50 and outputs a bit stream. The bit stream includes encoded image information and syntax.

  The transmission device 110 transmits a bit stream to a predetermined transmission path 120. The transmission path 120 is wired or wireless, and may be a communication line such as the Internet, a telephone line, or a radio wave for terrestrial broadcasting or satellite broadcasting that transmits a television signal. The receiving device 150 receives the bit stream transmitted through the transmission path 120. The image decoding apparatus 200 according to an embodiment decodes a bit stream and outputs decoded image information.

<Image encoding device>
FIG. 6 shows a specific configuration example of the image encoding device 100. In FIG. 6, the image information input to the image coding apparatus 100 is in a format including an allocation area where an image is allocated in a frame and a non-allocation area where no image is allocated, as shown in FIG. Image information.

  Pixels of image information of digital signals are sequentially input to the reorder buffer 1. If the image information is an analog signal, it may be converted into a digital signal by the A / D converter in the previous stage of the rearrangement buffer 1. The image information is, for example, a luminance signal Y (hereinafter referred to as a Y signal) and color difference signals Cb and Cr (hereinafter referred to as a Cb and Cr signal).

  The rearrangement buffer 1 accumulates input pixels for a plurality of frames, and rearranges and reads frames (pictures) as necessary. Each frame of image information is encoded using an I picture that is encoded using pixels in the frame, a P picture that is encoded predictively using pixels in a past frame, and a predictive encoding using pixels in past and future frames Is set to one of the B pictures. The I picture, P picture, and B picture may be set by the preprocessing device 50, or may be selected by the image encoding device 100 according to a predetermined rule.

  The rearrangement buffer 1 rearranges the order of frames for encoding when a plurality of picture groups (GOPs), which are constituent units of a sequence constituting a bit stream to be described later, are included. The pixels of each frame read from the rearrangement buffer 1 are supplied to the subtracter 2 via the picture dividing unit 104.

  Before describing the operations of the tile division setting unit 101, the skip tile setting unit 102, the reference tile setting unit 103, and the picture division unit 104, operations after the subtractor 2 will be described.

  Each frame input to the subtracter 2 is divided into, for example, horizontal 64 pixel and vertical 64 pixel CTUs. Each CTU is divided into a variable size coding unit (CU: Coding) based on recursive quadtree block division. Unit). The maximum size CU is referred to as a maximum coding unit (LCU), and the minimum size CU is referred to as a minimum coding unit (SCU).

  The CU is divided into prediction units (PUs) for prediction processing in the intra prediction unit 14 and the inter prediction unit 15. Further, the CU is divided into variable-size transform units (TUs) based on recursive quadtree block division for orthogonal transformation processing in the orthogonal transformation unit 3 and quantization processing in the quantization unit 4. Is done.

  The variable size CU, PU, and TU described above are selected so that the cost function value described later is minimized. Since the CU, PU, and TU that minimize the cost function value are different according to the pattern of the image information, the pixels of each frame are divided in a state where CU, PU, and TU having different sizes are mixed.

  The subtracter 2 subtracts a prediction value (predicted image) generated by the intra prediction unit 14 or the inter prediction unit 15 (to be described later) from the CU of the image information that is the original image output from the picture dividing unit 104 to perform prediction. Generate a residual. The subtracter 2 supplies the prediction residual to the orthogonal transformation unit 3.

  The orthogonal transform unit 3 performs orthogonal transform on the prediction residual in units of TUs to convert the prediction residual into a frequency domain signal. The orthogonal transform unit 3 uses the discrete sine transform (DST) only when the intra prediction is selected and the TU is 4 pixels horizontal and 4 pixels vertical, and the discrete cosine transform (DCT) is used in other cases. The orthogonal transformation is performed on the prediction residual. The orthogonal transform unit 3 supplies the orthogonal transform coefficient to the quantization unit 4.

  The quantization unit 4 quantizes the orthogonal transform coefficient and supplies it to the entropy coding unit 5 and the inverse quantization unit 8. The entropy encoding unit 5 assigns codes having different lengths to the quantized orthogonal transform coefficients based on the occurrence probabilities, and entropy codes the orthogonal transform coefficients.

  The entropy encoding unit 5 also entropy encodes the syntax for encoding image information. The syntax includes various syntax elements such as information for specifying a prediction mode, a motion vector, and a reference pixel selected by the intra prediction unit 14 or the inter prediction unit 15. The syntax also includes syntax elements indicating setting information in the tile division setting unit 101, the skip tile setting unit 102, and the reference tile setting unit 103.

  As an example, the entropy encoding unit 5 can entropy encode orthogonal transform coefficients and syntax using context adaptive arithmetic code (CABAC: Context-based Adaptive Binary Arithmetic Coding).

  The rate control unit 7 controls the rate of the quantization operation in the quantization unit 4 so that the encoded data output from the entropy encoding unit 5 does not overflow or underflow.

  An HRD (Hypothetical Reference Decoder) buffer 6 temporarily accumulates and outputs a bit stream composed of encoded data output from the entropy encoding unit 5.

  The inverse quantization unit 8 inversely quantizes the quantized orthogonal transform coefficient in units of TU and supplies the quantized orthogonal transform coefficient to the inverse orthogonal transform unit 9. The inverse quantization operation in the inverse quantization unit 8 is an operation opposite to the quantization operation in the quantization unit 4.

  The inverse orthogonal transform unit 9 performs inverse orthogonal transform on the input orthogonal transform coefficient in units of TU, and supplies the prediction residual to the adder 10. The adder 10 adds the input prediction residual and the prediction value generated by the intra prediction unit 14 or the inter prediction unit 15 selected by the prediction value selection unit 16 to generate a decoded signal. The decoded signal is supplied to the loop filter 11 and the frame memory 12.

  The loop filter 11 reduces coding noise of the decoded signal. The loop filter 11 includes a deblocking filter that reduces distortion generated at a block boundary and a pixel adaptive offset that reduces ringing distortion. The decoded signal filtered by the loop filter 11 is supplied to the frame memory 12.

  The frame memory 12 stores the decoded signal output from the adder 10 that has not been subjected to the filter processing by the loop filter 11 and the decoded signal that has been subjected to the filter processing by the loop filter 11. The switch 13 supplies the decoded signal stored in the frame memory 12 that has not been subjected to the filter processing to the intra prediction unit 14 and supplies the decoded signal stored in the frame memory 12 to which the filter processing has been performed to the inter prediction unit 15. To do.

  The intra prediction unit 14 generates predicted values in a plurality of prediction modes in units of TUs for each of the Y signal, the Cb signal, and the Cr signal. However, the prediction mode is selected in units of PUs. The intra prediction unit 14 calculates the cost function value, selects the size of the CU, PU, and TU that minimizes the cost function value, and selects the prediction value of the prediction mode that minimizes the cost function value.

  The inter prediction unit 15 detects the motion of the image, and sets the maximum size of the CU to a maximum of 64 horizontal pixels and 64 vertical pixels from a PU of 8 horizontal pixels, 4 vertical pixels or 4 horizontal pixels, and 8 vertical pixels. A prediction value is generated by performing inter-frame motion compensation prediction in the PU. The inter prediction unit 15 generates a predicted value (predicted image) with reference to a past frame, a future frame, or a past and future frame. Each of the past and future frames may be a plurality of frames.

  Similarly, the inter prediction unit 15 calculates the cost function value, selects the size of the CU, PU, and TU that minimizes the cost function value, and calculates the prediction value of the prediction mode that minimizes the cost function value. select.

  The prediction value selection unit 16 selects a subtraction unit having a smaller cost function value as a final prediction value among the prediction value selected by the intra prediction unit 14 and the prediction value selected by the inter prediction unit 15. 2 and the adder 10.

  Note that only the prediction value by the intra prediction unit 14 is used when encoding an I picture. When the P picture and the B picture are encoded, the smaller one of the prediction value selected by the intra prediction unit 14 and the prediction value selected by the inter prediction unit 15 is selected.

  The part from the subtractor 2 to the entropy encoding unit 5 and the part from the inverse quantization unit 8 to the predicted value selection unit 16 function as an encoded data generation unit that encodes image information and generates encoded data. To do.

  Next, operations of the tile division setting unit 101, the skip tile setting unit 102, the reference tile setting unit 103, and the picture division unit 104 will be described.

  The image information is also input to the tile division setting unit 101. The tile division setting unit 101 sets each frame to be divided into a plurality of tiles. The skip tile setting unit 102 sets a tile to be skipped. The reference tile setting unit 103 sets a tile to be referred to. Each piece of information set by the tile division setting unit 101 to the reference tile setting unit 103 is supplied to the picture division unit 104.

  The tile division setting unit 101 to the reference tile setting unit 103 may perform each setting under a preset condition, or may perform each setting according to an instruction from the preprocessing device 50.

  The image information supplied to the picture dividing unit 104 is divided into one or a plurality of slices. As shown in FIG. 7, a frame made up of a plurality of CTUs includes at least one slice. In FIG. 7, a thick solid line indicates a slice boundary, and the frame is divided into, for example, three slices SL0 to SL2. How the frame is divided into a plurality of slices is set by the preprocessing device 50. Each of the slices SL0 to SL2 includes at least one continuous CTU.

  The image encoding device 100 encodes image information in units of slices, and the image decoding device 200 decodes image information in units of slices. In FIG. 7, arrows indicated by solid lines indicate the order of encoding and decoding.

  Also, a frame composed of a plurality of CTUs can be divided into a plurality of tiles by setting by the tile division setting unit 101 as shown in FIG. In FIG. 8, thick solid lines indicate tile boundaries, and the frame is divided into six tiles TL0 to TL8. As will be described later, although there is a restriction on the relationship between the slice and the tile, here, the division of the tile is shown regardless of the division of the slice shown in FIG.

  The image encoding device 100 encodes image information in tile units, and the image decoding device 200 decodes image information in tile units. In FIG. 8, arrows indicated by solid lines indicate the order of encoding and decoding. The tile can be used for parallel processing or ROI (Region Of Interest) coding in addition to coding of a VR image.

  FIG. 9A shows a state in which slices SL0 and SL1 are set in the tile TL0, and slices SL2 and SL3 are set in the tile TL1. A tile can be a superset of slices. FIG. 9B shows a state where tiles TL0 and TL1 are set in the slice SL0. A slice can be a superset of tiles. As shown in FIGS. 9A and 9B, the relationship between slices and tiles can be set.

  The slices SL0 to SL2 and the tiles TL0 and TL1 in FIG. 9C cannot be set because one of the slices and tiles is not a superset and the other is not a subset.

  In FIG. 6, the tile division setting unit 101 sets tile division for each frame specifically as follows. The tile division setting unit 101 sets the number of tiles included in one frame. Assume that the number of tiles is 4, and four tiles are set as shown in FIG.

  In FIG. 10, tiles TL0 to TL3 with tile numbers 0 to 3 can be defined as shown in FIG. The hatched tiles TL1 and TL3 correspond to the non-allocation areas in FIG. The tiles TL0 and TL2 that are not hatched are assigned areas to which images are assigned.

  As shown in FIG. 11, the tile TL0 with tile number 0 is defined as a horizontal size h0 and a vertical size v0 with the coordinates of the upper left corner indicated by a black circle as the horizontal position X0 and the vertical position Y0. The skip tile flag in the tile TL0 is 0, indicating that the tile is not skipped.

  The tile TL1 with the tile number 1 is defined as a horizontal size h1 and a vertical size v1 with the coordinates of the upper left corner indicated by a black circle as the horizontal position X1 and the vertical position Y1. The skip tile flag in the tile TL1 is 1, indicating that the tile is skipped.

  The tile TL2 of tile number 2 is defined as a horizontal size h1 and a vertical size v2 with the coordinates of the upper left corner indicated by a black circle as the horizontal position X2 and the vertical position Y2. The skip tile flag in the tile TL1 is 0. The tile TL3 with the tile number 3 is defined as a horizontal size h1 and a vertical size v3 with the coordinates of the upper left corner indicated by a black circle as the horizontal position X3 and the vertical position Y3. The skip tile flag in the tile TL3 is 1.

  The horizontal and vertical positions of the tiles TL0 to TL3 are coordinates in CTU units. The horizontal size and vertical size of the tiles TL0 to TL3 are CTU units. For example, if the size of the CTU is 64 pixels horizontal and 64 pixels vertical, the horizontal size is 128 pixels, and the vertical size is 192 pixels, the syntax element indicating the transmitted horizontal size is 2, and the syntax element indicating the vertical size is 2. The tax element is 3. In this embodiment, the number of tiles included in one frame is set, and the coordinates of the upper left corner of each tile, the horizontal size, and the vertical size are set, which cannot be set in the conventional HEVC. Tiles can be set.

  The skip tile setting unit 102 can set tiles to be skipped that cannot be set by the conventional HEVC. In FIG. 10, if the tiles TL1 and TL3 are set to be skipped tiles, it is not necessary to encode the tiles TL1 and TL3, and the encoding efficiency can be improved.

  A tile TL4 and a tile TL7 illustrated in FIG. 12 are images projected on adjacent surfaces (402F and 402A) in the hexahedron 402 illustrated in FIG. 1 although they are separated in the frame. Therefore, the tile TL4 and the tile TL7 have a high correlation. Therefore, the reference tile setting unit 103 can set one or a plurality of tiles to be referenced for each tile. The reference tile setting information generated by the reference tile setting unit 103 is information indicating a prediction dependency relationship between a certain tile to be encoded and a reference tile.

  Returning to FIG. 6, the picture dividing unit 104 divides each frame based on the tile division setting information set as shown in FIG. The picture dividing unit 104 divides the allocated area into one or more tiles (first tile) and divides the non-allocated area into one or more tiles (second tile) according to the format of the image information. That's fine. In the format shown in FIG. 2, it is necessary to divide the allocation area and the non-allocation area into a plurality of tiles.

  The picture dividing unit 104 supplies the tile division, the tile to be skipped, and the setting information of the tile to be referenced to the entropy encoding unit 5. The entropy encoding unit 5 generates a syntax including syntax elements indicating the setting information and performs entropy encoding. The entropy encoding unit 5 functions as a syntax generation unit and a bit stream generation unit.

  FIG. 13 shows an example of a syntax including syntax elements related to tiles generated and output by the entropy encoding unit 5. The syntax element related to the tile may be transmitted by being included in the picture parameter set (Picture Parameter Set) of the bit stream.

  When the tile enable flag (tiles_enabled_flag) is 1, the syntax includes horizontal position (hpos), vertical position (vpos), horizontal size (hsize), vertical size (vsize), skip tile flag (skip_tile_flag), number of reference tiles ( num_ref_tiles) and reference tile number (ref_tile_no) are set.

  The skip tile flag 0 indicates that the tile is not skipped, and the skip tile flag 1 indicates that the tile is skipped. As shown in FIG. 13, the number of reference tiles may be zero. Note that loop_filter_across_tiles_enabled_flag in FIG. 13 is a flag for setting whether to perform filtering by the loop filter 11 between tiles.

  As described above, the image encoding device 100 improves the encoding efficiency of a VR image in a format including an allocation area to which an image is allocated in a frame and a non-allocation area to which no image is allocated. The amount of processing can be reduced.

<Image coding method>
The process by the image coding method of this embodiment performed with the image coding apparatus 100 is demonstrated using the flowchart shown in FIG. FIG. 14 shows processing executed by the tile division setting unit 101, the skip tile setting unit 102, and the reference tile setting unit 103.

  In FIG. 14, the tile division setting unit 101 sets the number of tiles in step S101, and first sets the tile number to 0 in step S102. The tile division setting unit 101 sets a horizontal position and a vertical position in step S103, and sets a horizontal size and a vertical size in step S104.

  In step S105, the skip tile setting unit 102 sets a skip tile flag for each tile. In step S106, the reference tile setting unit 103 determines whether or not the skip tile flag is 1. If the skip tile flag is not 1 (NO), the reference tile setting unit 103 sets the reference tile number in step S107. The number of reference tiles is a number of 0 or more, and 0 indicates that other tiles are not referred to, and a number of 1 or more indicates that the number of tiles is referred to.

  In step S108, the reference tile setting unit 103 determines whether the reference tile number is other than zero. If the reference tile number is other than 0 (YES), the reference tile setting unit 103 sets a reference tile number in step S109. Thereafter, the process proceeds to step S110.

  If the skip tile flag is 1 in step S106 (YES), and if the number of reference tiles is not 0 in step S108 (NO) (that is, if the number of reference tiles is 0), the process proceeds to step S106. Moved to S110.

  In step S110, the tile division setting unit 101 determines whether the tile number is equal to the number of tiles -1. If the tile number is not -1 (NO), since tiles still remain, the tile division setting unit 101 increments the tile number by 1 in step S111 and returns the process to step S103.

  The tile division setting unit 101, the skip tile setting unit 102, and the reference tile setting unit 103 repeat the processes of steps S103 to S110 for the tile with the next tile number. If the tile number is -1 in step S110 (YES), the setting for all tiles has been completed, so the tile division setting unit 101, the skip tile setting unit 102, and the reference tile setting unit 103 end the processing. .

<Image coding program>
The operation of the image encoding device 100 shown in FIG. 6 can be executed by a computer by a computer program (image encoding program). Each process shown in FIG. 14 can be executed by a computer using an image encoding program.

  In FIG. 15A, the computer 300 includes a central processing unit (CPU) 301 and a storage unit 302. An operation unit 310 is connected to the computer 300. The storage unit 302 stores an image encoding program. The operation unit 310 can function as the preprocessing device 50 shown in FIG. By causing the CPU 301 to execute the image encoding program, the computer 300 can function as the image encoding device 100 that encodes the input image information.

  The storage unit 302 is an arbitrary non-temporary storage medium such as a semiconductor memory, a hard disk drive, or an optical disk. The image encoding program may be provided to the computer 300 via a communication line such as the Internet.

  The image encoding device 100, the image encoding method, and the image encoding program described above encode image information so that tiles do not overlap and there is no area in which no tile is set in one frame. It shall be.

<Image decoding device>
FIG. 16 shows a specific configuration example of the image decoding device 200. In FIG. 16, the HRD buffer 21 temporarily accumulates the bit stream and supplies it to the entropy decoding unit 22. The HRD buffer 21 functions as a bit stream receiving unit that receives a bit stream.

  The bit stream includes encoded data and syntax. As described above, the encoded data is encoded data obtained by encoding image information in a format including an allocated area in which an image is allocated in a frame and an unallocated area in which no image is allocated.

  Also, the encoded data is divided into one or a plurality of tiles set in the allocation area and one or a plurality of tiles set in the non-allocation area, and an image of the tile set in the allocation area. Is the encoded data.

  The syntax includes syntax elements that define each tile with a horizontal position and vertical position of the upper left corner in the frame, and a horizontal size and vertical size. It is preferable that the syntax includes a syntax element that is a skip tile flag indicating that the image of the tile set in the non-allocation area is not encoded. The syntax preferably includes a syntax element indicating a prediction dependency.

  The entropy decoding unit 22 entropy decodes the orthogonal transform coefficients and syntax included in the bitstream.

  The orthogonal transform coefficient is supplied to the inverse quantization unit 23. Information indicating whether intra prediction or inter prediction is adopted is supplied to the switch 32. Information regarding intra prediction is supplied to the intra prediction unit 30. Information regarding inter prediction is supplied to the inter prediction unit 31.

  Information relating to tile division is supplied to the tile division decoding unit 201. Information regarding the skip tile is supplied to the skip tile setting unit 202 via the tile division decoding unit 201. Information on the reference tile is supplied to the reference tile setting unit 203 via the inter prediction unit 31, the tile division decoding unit 201, and the skip tile setting unit 202.

  The inverse quantization unit 23 inversely quantizes the orthogonal transform coefficient in units of TU and supplies the inverse transform coefficient to the inverse orthogonal transform unit 24. The inverse orthogonal transform unit 24 performs inverse orthogonal transform on the inversely quantized orthogonal transform coefficient in units of TU, and supplies the prediction residual to the adder 25. The adder 25 adds the input prediction residual and the prediction value generated by the intra prediction unit 30 or the inter prediction unit 31 supplied from the switch 32 to generate a decoded signal. The decoded signal is supplied to the loop filter 26 and the frame memory 28.

  The loop filter 26 has the same configuration as the loop filter 11 and reduces the encoding noise of the decoded signal. The re-order buffer 27 accumulates pixels supplied from the loop filter 26 for a plurality of frames. The rearrangement buffer 27 rearranges the frames in the order of the image information of the original image if the order of the frames is rearranged. The pixels constituting the tile of each frame output from the rearrangement buffer 27 are supplied to the picture composition unit 204.

  The frame memory 28 stores the decoded signal output from the adder 25 that has not been subjected to the filter processing by the loop filter 26 and the decoded signal that has been subjected to the filter processing by the loop filter 26. The switch 29 supplies the decoded signal that has not been subjected to the filter processing accumulated in the frame memory 28 to the intra prediction unit 30, and supplies the decoded signal that has been subjected to the filter processing accumulated in the frame memory 28 to the inter prediction unit 31. To do.

  The intra prediction unit 30 performs intra-frame prediction according to information indicating a prediction mode of intra prediction, and generates respective prediction values of the Y signal, the Cb signal, and the Cr signal. The inter prediction unit 31 performs inter-frame prediction in accordance with information related to inter prediction, and generates respective prediction values for the Y signal and the Cb and Cr signals.

  When the information regarding the reference tile is input, the inter prediction unit 31 refers to an image of another tile (here, TL7) and predicts the image of a certain tile (here, TL4) as illustrated in FIG. Is generated.

  The switch 32 supplies the adder 25 with the prediction value generated by the intra prediction unit 30 or the inter prediction unit 31 according to information indicating which of intra prediction and inter prediction is adopted.

  The portion from the entropy decoding unit 22 to the switch 32 functions as a decoding unit that decodes the encoded data and generates a decoded image of the tile set in the allocation area.

  The tile division decoding unit 201 decodes the tile division setting information set as illustrated in FIG. 11 and supplies the decoded information to the skip tile setting unit 202. The skip tile setting unit 202 sets a tile to be skipped based on the skip tile flag. The reference tile setting unit 203 sets a tile to be referred to. The tile division setting information, the tile information to be skipped, and the tile information to be referred to are supplied to the picture composition unit 204.

  The picture composition unit 204 arranges the decoded tile image in the allocation area in the frame based on the tile division setting information. The picture composition unit 204 arranges, for example, a single black image in the non-allocation area based on the information on the tile to be skipped. The picture composition unit 204 synthesizes a plurality of tiles in this way, and generates decoded image information of each frame of the decoded image information in the format shown in FIG. The decoded image information is converted into an analog signal by a D / A converter as necessary.

  As described above, the image decoding apparatus 200 can reduce the amount of processing for decoding a VR image having a format including an allocation area to which an image is allocated in a frame and a non-allocation area to which no image is allocated. it can.

<Image decoding method>
The process by the image decoding method of this embodiment performed with the image decoding apparatus 200 is demonstrated using the flowchart shown in FIG. FIG. 17 illustrates processing executed by the tile division decoding unit 201, the skip tile setting unit 202, and the reference tile setting unit 203.

  In FIG. 17, the tile division decoding unit 201 receives the number of tiles in step S201, and first sets the tile number to 0 in step S202. The tile division decoding unit 201 receives the horizontal position and the vertical position in step S203, and receives the horizontal size and the vertical size in step S204.

  In step S205, the skip tile setting unit 202 receives the skip tile flag of each tile. In step S206, the reference tile setting unit 203 determines whether or not the skip tile flag is 1. If the skip tile flag is not 1 (NO), the reference tile setting unit 203 receives the reference tile number in step S207.

  In step S208, the reference tile setting unit 203 determines whether the reference tile number is other than zero. If the number of reference tiles is other than 0 (YES), the reference tile setting unit 203 receives the reference tile number in step S209. Thereafter, the process proceeds to step S210.

  If the skip tile flag is 1 in step S206 (YES), and if the reference tile number is not 0 in step S208 (NO) (that is, if the reference tile number is 0), the process proceeds to step S206. Moved to S210.

  In step S210, the tile division decoding unit 201 determines whether or not the tile number is equal to the number of tiles−1. If the tile number is not the number of tiles -1 (NO), since tiles still remain, the tile division decoding unit 201 increments the tile number by 1 in step S211 and returns the process to step S203.

  The tile division decoding unit 201, the skip tile setting unit 202, and the reference tile setting unit 203 repeat the processes in steps S203 to S210 for the tile with the next tile number. If the tile number is the number of tiles −1 in step S210 (YES), the tile division decoding unit 201, the skip tile setting unit 202, and the reference tile setting unit 203 end the processing because information on all tiles has been received. .

<Image decoding program>
The operation of the image decoding apparatus shown in FIG. 16 can be executed by a computer using a computer program (image decoding program). Each process shown in FIG. 17 can be executed by a computer using an image decoding program.

  As shown in FIG. 15B, the computer has the same configuration as that in FIG. 15A, and the description of the common parts with FIG. In FIG. 15B, the storage unit 302 stores an image decoding program instead of the image encoding program. By causing the CPU 301 to execute the image decoding program, the computer 300 can function as the image decoding device 200 that decodes the encoded bit stream.

  It is also possible to store the image encoding program and the image decoding program in the storage unit 302 so that the computer 300 functions as the image encoding device 100 and the image decoding device 200.

<First Modification>
In the image encoding device, the image encoding method, and the image encoding program, and the image decoding device, the image decoding method, and the image decoding program of the present embodiment described above, the following first modified example is used. be able to.

  As shown in FIG. 18, the order of encoding and transmitting the tiles does not have to be the top left tile first. In FIG. 18, the image encoding device 100 encodes the tiles TL <b> 0 to TL <b> 5 in this order and transmits a bit stream including encoded data. The image decoding apparatus 200 decodes in the order of the tiles TL0 to TL5. A tile TL0 that first encodes and decodes a tile corresponding to the field of view that the user wearing the head-mounted display first visually recognizes.

  The skip tile flag need not be transmitted. In this case, in the above description, the image information is encoded so that there is no region in which no tile is set in one frame, but there may be a region in which no tile is set. In FIG. 18, it is not necessary to set a tile in a non-allocated area with hatching.

<Second Modification>
As described with reference to FIG. 9B, the slice can be a superset of tiles, but the slice may be prohibited from being a superset of tiles. If the slice is not prohibited from being a superset of tiles, a slice SL0 that is a superset of tiles TL0 and TL1 that are located at a distance from each other may be set as shown in FIG. It is advisable to prohibit a slice from being a superset of tiles so that multiple tiles that are far from each other do not share the parameters contained in the slice header.

<Third Modification>
In the above description, tiles do not overlap in one frame, but tile overlap may be allowed. FIG. 20 shows a state where the tile TL0 indicated by the solid line and the tile TL1 indicated by the alternate long and short dash line overlap.

  When the field of view of the user wearing the head mounted display is in the area AR0 indicated by the broken line, the image decoding apparatus 200 decodes the image of the area AR0 in the tile TL0 and displays it on the head mounted display. When the user's field of view moves from the area AR0 to the area AR1 indicated by the broken line, the image decoding apparatus 200 overlaps the tile TL1 of the tile TL0 before starting the decoding process of the tile TL1. Decode the image and display it. Accordingly, it is possible to reduce a delay in image display when the user's visual field moves.

  In the present embodiment described above, Y, Cb, and Cr video signals are taken as examples of image information. However, the color space is not limited to Y, Cb, and Cr, and may be Y, Co, and Cg video signals. The image information may be a video signal in an arbitrary color space including a luminance component and a color component composed of two color difference signals.

  The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the scope of the present invention. The image encoding device 100 and the image decoding device 200 may be configured by hardware such as an integrated circuit, may be configured by software, or both may be mixed. The image encoding method and the image decoding method may be executed by any hardware resource such as an integrated circuit or a computer.

DESCRIPTION OF SYMBOLS 100 Image coding apparatus 101 Tile division | segmentation setting part 102,202 Skip tile setting part 103,203 Reference tile setting part 104 Picture division part 200 Image decoding apparatus 201 Tile division decoding part 204 Picture composition part 300 Computer 302 Storage part

Claims (10)

Image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated is input, and each frame of the image information is set to one or more set in the allocation area A picture dividing unit for dividing the first tile into one or a plurality of second tiles set in the non-allocation area;
An encoded data generation unit that encodes the image of the first tile to generate encoded data;
Syntax generation for generating a syntax including a first syntax element for each of the first and second tiles defined by the horizontal position and vertical position of the upper left corner in the frame, and the horizontal size and vertical size. And
A bit stream generation unit that generates a bit stream including the encoded data and the syntax;
An image encoding device comprising: The encoded data generation unit does not encode the image of the second tile,
The syntax generation unit generates a syntax including a second syntax element that is a skip tile flag indicating that the image of the second tile is not encoded. Image encoding device. The picture dividing unit divides the allocation area into a plurality of first tiles,
The encoded data generation unit is based on setting information on whether or not to refer to at least one other tile of the plurality of first tiles when encoding each of the plurality of first tiles. And encoding each of the plurality of first tiles to generate encoded data,
The syntax generation unit indicates that when the encoded data generation unit encodes the first tile without referring to other tiles, the syntax generation unit does not refer to other tiles, and refers to the other tiles. The image encoding apparatus according to claim 2, wherein when one tile is encoded, a syntax including a third syntax element indicating a tile to be referred to is generated. Each frame of image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and one or more first tiles set in the allocation area; Dividing into one or a plurality of second tiles set in the non-allocation area,
Encoding the image of the first tile to generate encoded data;
Each of the first and second tiles generates a syntax including syntax elements defined by a horizontal position and a vertical position of the upper left corner in the frame, and a horizontal size and a vertical size,
An image encoding method, comprising: generating a bitstream including the encoded data and the syntax. On the computer,
Each frame of image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and one or more first tiles set in the allocation area; Dividing into one or more second tiles set in the unallocated area;
Encoding the image of the first tile to generate encoded data;
Each of the first and second tiles generating a syntax including syntax elements defined by a horizontal position and a vertical position of the upper left corner in the frame and a horizontal size and a vertical size;
Generating a bitstream including the encoded data and the syntax;
An image encoding program characterized in that It is image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and each frame of the image information includes one or a plurality of frames set in the allocation area Divided into a first tile and one or a plurality of second tiles set in the non-allocation area, encoded data obtained by encoding an image of the first tile, and the first and second A bit stream receiving unit that receives a bit stream including a syntax including a first syntax element defined by a horizontal position and a vertical position of the upper left corner in the frame and a horizontal size and a vertical size of each tile of When,
A decoding unit that decodes the encoded data to generate a decoded image of the first tile;
A picture synthesis unit that synthesizes the decoded image of the first tile and the single image of the second tile based on the first syntax element to generate decoded image information of each frame;
An image decoding apparatus comprising: The encoded data is encoded data in which the image of the second tile is not encoded,
The bitstream receiving unit receives a syntax including a second syntax element that is a skip tile flag indicating that the second tile is not encoded;
The image decoding apparatus according to claim 6, wherein the picture composition unit assigns the single image to the second tile based on the second syntax element. The allocation area of the image information is divided into a plurality of first tiles;
The bitstream reception unit indicates whether each of the plurality of first tiles is encoded with reference to at least one other tile of the plurality of first tiles, Receiving a syntax including a third syntax element indicating the referenced tile when encoded with reference to
The decoding unit decodes each of the plurality of first tiles based on the third syntax element without referring to the other tiles when not encoded with reference to the other tiles, The image decoding apparatus according to claim 7, wherein when decoding is performed with reference to another tile, the decoding is performed with reference to the other tile. It is image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and each frame of the image information includes one or a plurality of frames set in the allocation area Divided into a first tile and one or a plurality of second tiles set in the non-allocation area, encoded data obtained by encoding an image of the first tile, and the first and second Each of the tiles receives a bitstream including a syntax including a syntax element defined by a horizontal position and a vertical position of the upper left corner in the frame and a horizontal size and a vertical size;
Decoding the encoded data to generate a decoded image of the first tile;
Based on the syntax element, the decoded image information of each frame is generated by synthesizing the decoded image of the first tile and the single image of the second tile. On the computer,
It is image information in a format including an allocation area in which an image is allocated in a frame and a non-allocation area in which no image is allocated, and each frame of the image information includes one or a plurality of frames set in the allocation area Divided into a first tile and one or a plurality of second tiles set in the non-allocation area, encoded data obtained by encoding an image of the first tile, and the first and second Receiving a bitstream that includes a syntax including a syntax element defined by a horizontal position and a vertical position of the upper left corner and a horizontal size and a vertical size of each of the tiles in the frame;
Decoding the encoded data to generate a decoded image of the first tile;
Combining the decoded image of the first tile and the single image of the second tile based on the syntax element to generate decoded image information of each frame;
An image decoding program characterized in that

Images