JP2019129411A - Picture encoder, imaging apparatus, image coding method, and program - Google Patents

Abstract

To provide a technique for restraining increase in the time required for encoding, when filtering the boundary region of the division regions of the picture.SOLUTION: A picture encoder for encoding multiple pictures included in a dynamic picture image according to a prescribed coding order includes division means for dividing a first image included in the dynamic picture image into multiple division regions, coding means for encoding the first image in units of division region, and encoding a second image included in the dynamic picture image by using motion compensation interframe prediction, decoding the encoded first image, and filter means for filtering the boundary region of the multiple division regions in the decoded first image. When the first image is located just before the second image in a prescribed coding order, and the first image is used as a reference image for motion compensation interframe prediction, the coding means excludes the boundary region of the multiple division regions in the first image from the search range of motion compensation interframe prediction.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image coding apparatus, an imaging apparatus, an image coding method, and a program.

  The HEVC (High Efficiency Video Coding) coding system, which is an international standard coding standard for moving images, has begun to spread. In the HEVC coding scheme, the conventional H. A technique called tile division is newly added to the H.264 encoding method (see Non-Patent Document 1). In this method, a screen is divided into a plurality of divided areas called tiles, and encoding operations are performed in order for each tile. For example, when the screen is divided into two tiles on the left and right, the encoding operation of the left tile is performed first, and then the encoding operation of the right tile is performed. In the normal encoding operation where tile division is not performed, encoding is performed in raster order for each block from the upper left corner of one screen, but in tile division encoding operation, encoding is performed sequentially for each tile, and blocks are arranged from the upper left corner in the tile. Encode in raster order for each Therefore, the coding order is different compared to the case where tile division is not performed.

  Also in the HEVC coding method, H.1. A process called a deblocking filter is performed similarly to the H.264 encoding method. This is a filtering process for removing block noise on block boundaries. Usually, in the case of encoding operation in raster order on one screen without tile division, a deblocking filter process is performed for each block. The deblocking filter is filtered at upper and lower block boundaries and at left and right block boundaries. Therefore, the encoder normally holds pixels after local decoding near the block boundary for filtering. In the case of processing in raster order, pixels for filters at the left and right boundaries may be held for each block. With regard to the pixels for the filter of the upper and lower boundaries, it is sufficient to hold pixels of a plurality of lines for the filtering process of the upper boundary.

  In the case of tile division processing, the encoding order is different from normal. In this case, there is no problem with the deblocking filter process in the tile, but the deblocking filter process at the tile boundary becomes a problem. An example is shown in FIG. First, the encoder performs an encoding process on the left tile 201 and also performs a deblocking filter process. Next, the encoder performs encoding processing of the right tile 202. Here, since the encoding order is different from the case where tile division is not performed, when the deblocking filter processing is performed on the tile division boundary 203, the block previously encoded is deblocked at the tile division boundary 203. There are no pixels for filtering. Therefore, in order to enable the deblocking filter process for the tile division boundary 203, some configuration is required that enables the pixels for the deblocking filter process on the tile division boundary 203 to be acquired.

  In Patent Document 1, in addition to the pixels conventionally held for the deblocking filter, all pixels in the vicinity of the left and right tile division boundaries (hereinafter also referred to as “tile boundaries”) are retained for the vertical resolution. It discloses technology. According to the technique of Patent Document 1, all the pixels corresponding to the resolution in the vertical direction in the vicinity of the tile boundary are held, so that a large amount of buffer is required.

  Alternatively, a configuration may be considered in which only the deblocking filter is performed on tile boundaries after all encoding processing other than the deblocking filter processing on tile boundaries is performed on the entire screen. That is, the encoder performs encoding processing other than tile block deblocking filter processing, temporarily stores one screen's worth of locally decoded images, and then performs tile block deblocking processing. Processing will be performed. In this case, since it is not necessary to hold vertical pixels in the vicinity of the boundary for deblocking filtering of the tile boundary, the required buffer size can be reduced.

JP 2017-50766 A

H. 265 / HEVC textbook, Satoshi Okubo [supervised], Teruhiko Suzuki, ▲ Takamura Mura, Ken Nakajo [edit], published on October 21, 2013

  When the processing is divided into two stages as described above, the processing time is long. In order to reduce the processing time, it is conceivable to perform two processes in parallel for each picture. That is, encoding processing other than tile block deblocking filtering and tile block deblocking filtering may be performed by pipeline processing in units of pictures. However, in order to perform encoding using the locally decoded image that has been encoded immediately before as a reference image, it is necessary to wait for tile boundary deblocking filter processing to be completed. Therefore, such pipeline processing cannot be applied, for example, when encoding a B picture immediately after a P picture.

  The present invention has been made in view of such a situation, and a technique for suppressing an increase in time required for encoding when filtering is performed on a boundary region of divided regions in an image encoded in units of divided regions. Intended to provide.

  In order to solve the above problems, the present invention is an image coding apparatus that codes a plurality of images included in a moving image according to a predetermined coding order, and a plurality of first images included in the moving image A dividing unit that divides the first image into division regions, an encoding unit that encodes the first image in units of divided regions, and encodes a second image included in the moving image using motion compensation interframe prediction; Decoding means for decoding the encoded first image, and filter means for performing filtering on boundary regions of the plurality of divided regions in the decoded first image, Is located immediately before the second image in the predetermined encoding order, and the first image is used as a reference image for the motion compensation inter-frame prediction, the encoding means includes: In the first image To provide an image coding apparatus characterized by excluding the boundary region of the serial plurality of divided regions from the search range of the prediction between the motion compensated frame.

  According to the present invention, it is possible to suppress an increase in time required for encoding when filtering is performed on a boundary region of divided regions in an image encoded in units of divided regions.

  Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following description of the preferred embodiments.

FIG. 2 is a block diagram illustrating the configuration of an imaging device 100. FIG. 7 is a diagram for describing an encoding operation involving tile division. The figure which shows the input order of an encoding object image, an encoding order, and a reference relationship. FIG. 7 is a diagram for describing an encoding operation in the case where limitation of a search range for motion vector detection is not performed. The figure which shows the search range of the reference image which the image 304 (FIG. 4) refers. The figure explaining the encoding operation | movement of the image 306,304,305,309 (FIG. 4) in the case of restrict | limiting a search range. 6 is a flowchart of encoding processing performed by the imaging device 100. FIG. 10 is a block diagram for explaining the arrangement of an imaging apparatus 800. FIG. 10 is a block diagram for explaining the arrangement of an imaging apparatus 900. The figure explaining the detail of the image coding part 902 (FIG. 9). 12 is a flowchart of encoding processing performed by the imaging device 900. The figure explaining the example of the encoding operation | movement according to the flowchart of FIG. 10 is a flowchart of an entropy coding process performed by the imaging device 900. FIG. 14 is a view for explaining an example of the entropy coding operation according to the flowchart of FIG. 13;

  Embodiments of the present invention will be described below with reference to the accompanying drawings. Throughout the attached drawings, elements given the same reference numerals represent the same or similar elements. The technical scope of the present invention is determined by the scope of the claims, and is not limited by the following individual embodiments. Moreover, not all combinations of features described in the embodiments are essential to the present invention. Moreover, it is possible to appropriately combine the features described in different embodiments.

First Embodiment
FIG. 1 is a block diagram for explaining the arrangement of an imaging apparatus 100 including the image coding apparatus according to the first embodiment. Although the example which encodes a moving image by a HEVC encoding system is demonstrated below, the encoding system of this embodiment is not limited to a HEVC encoding system.

  First, a normal encoding operation that does not perform tile division will be described. A light flux from a subject is input to the imaging unit 102 through the lens 101. The imaging unit 102 converts the input light flux into digital pixel data, and outputs the digital pixel data to the development processing unit 103.

  The development processing unit 103 performs various types of image processing such as debayering processing, flaw correction, noise removal, and color conversion to the YCbCr format. After image processing, an image in a format that can be compressed and encoded is input to the encoding frame buffer 104 as an encoding target image to be encoded.

  The motion prediction unit 105 performs block matching between the encoding target image stored in the encoding frame buffer 104 and the reference image stored in the reference frame buffer 116, and performs motion vector detection. At this time, the reference image range setting unit 118 sets a search range in which motion vector detection is performed. The motion prediction unit 105 takes a pixel difference between the encoding target image and the predicted image at the detected motion vector position, and outputs the difference image to the orthogonal transformation unit 106. Also, the motion prediction unit 105 outputs the predicted image to the motion compensation unit 113 for local decoded image creation.

  The orthogonal transform unit 106 generates a transform coefficient by performing discrete cosine transform on the difference image, and outputs the transform coefficient to the quantization unit 107.

  The quantization unit 107 quantizes the transform coefficient sent from the orthogonal transform unit 106 in accordance with the quantization step (quantization value) output from the quantization control unit 108. The quantized transform coefficients are output to the variable-length coding unit 109 to generate a coded stream. Also, the quantized transform coefficients are output to the inverse quantization unit 111 in order to create a local decoded image.

  The variable-length coding unit 109 performs, for example, zigzag scan or alternate scan on the quantized transform coefficients to perform variable-length coding. The variable-length coding unit 109 performs variable-length coding on coding method information such as motion vectors, quantized values, macroblock division information, and adaptive offset processing parameters. The variable-length coding unit 109 generates a coded stream by adding coded coding method information to the coded transform coefficient. The generated encoded stream is stored in the recording medium 110. In addition, the variable length coding unit 109 calculates a generated code amount for each macroblock at the time of coding, and outputs it to the quantization control unit 108.

  The quantization control unit 108 determines the quantization value so as to achieve the target code amount using the generated code amount sent from the variable length coding unit 109, and outputs the quantization value to the quantization unit 107.

  The inverse quantization unit 111 performs inverse quantization on the quantized transform coefficient sent from the quantization unit 107 to generate a transform coefficient for local decoding. The transform coefficients are output to the inverse orthogonal transform unit 112.

  The inverse orthogonal transform unit 112 generates a difference image by performing inverse discrete cosine transform on the transform coefficient output from the inverse quantization unit 111. The generated difference image is output to the motion compensation unit 113.

  The motion compensation unit 113 generates the image data for local decoding by adding the motion vector position predicted image sent from the motion prediction unit 105 and the difference image sent from the inverse orthogonal transform unit 112. Do. The generated image data is output to the deblocking filter unit 114.

  The deblocking filter unit 114 applies a deblocking filter to the image data sent from the motion compensation unit 113. The image data after the deblocking filter is output to the adaptive offset processing unit 115.

  The adaptive offset processing unit 115 selects one of band offset processing, edge offset processing, or no processing, and determines a band position, an edge direction, an offset value, and the like on which the adaptive offset processing is performed. The adaptive offset processing unit 115 performs adaptive offset processing on the image data after the deblocking filter, and stores the image after the adaptive offset processing in the reference frame buffer 116 as a local decoded image. In addition, the adaptive offset processing unit 115 has a variable length in order to include parameters for adaptive offset processing (information on which processing is selected as adaptive offset processing, band position, edge direction, offset value, etc.) in the encoded stream. Output to encoding section 109.

  Through the above processing, an encoded stream and a local decoded image are created.

  Next, an encoding operation involving tile division will be described. Here, a case will be described in which the encoding target image is divided into two left and right tiles (divided regions) as shown in FIG. First, for the left tile 201, the encoding operation is performed in raster order within the tile, similarly to the normal encoding operation described above. When the encoding operation of tile 201 is completed, the encoding operation is similarly performed on the right tile 202. However, the deblocking filter process is not performed on the tile division boundary 203 during the encoding operation of the tile 202.

  Note that neither the tile 201 nor the tile 202 is subjected to adaptive offset processing in a macroblock (CTU) including pixels of the tile division boundary 203. This is because the adaptive offset process needs to be performed on the pixel after the deblocking filter process.

  After coding processing for the entire picture is completed and the local decoded image is stored in the reference frame buffer 116, deblocking filtering of the tile division boundary 203 is performed. Specifically, the tile boundary deblocking filter unit 117 reads, from the reference frame buffer 116, pixels necessary for the deblocking filter processing of the tile division boundary 203. Then, the tile boundary deblocking filter unit 117 performs the deblocking filter process on the tile division boundary 203 and stores the pixel after the deblocking filter process in the same location in the reference frame buffer 116.

  FIG. 3 is a diagram illustrating the input order of the encoding target image, the encoding order, and the reference relationship. In FIG. 3, images 301 to 310, which are sequentially input, are shown as images to be encoded. Encoding is performed by changing the order of these images. After the image 303 is input, the image 303 is first encoded as an I picture. Since the image 303 is an I picture, encoding is performed only in the screen. Next, images 301 and 302 are encoded as B pictures. The images 301 and 302 are encoded using motion compensation inter-frame prediction, and a locally decoded image of the image 303 is used as a reference image. Next, the image 306 is encoded as a P picture. The image 306 is encoded using motion compensation inter-frame prediction, and a local decoded image of the image 303 is used as a reference image. Next, images 304 and 305 are encoded as B pictures. The images 304 and 305 are encoded using motion compensation inter-frame prediction, and the local decoded images of the images 303 and 306 are used as reference images. Next, the image 309 is encoded as a P picture. The image 309 is encoded using motion compensation inter-frame prediction, and the local decoded image of the image 306 is used as a reference image.

  Here, with reference to FIG. 4, description will be given of an encoding operation when the search range for performing motion vector detection is not limited (details will be described later). In FIG. 4, the encoding operation of the images 306, 304, 305, and 309 includes encoding processing (Enc processing) other than deblocking filtering processing of tile boundaries and deblocking filtering processing (Deb processing) of tile boundaries. It is shown separately.

  First, the Enc processing 401 of the image 306 (P picture) is performed with reference to the local decoded image of the image 303. Thereafter, Deb processing 402 of the image 306 is performed. Next, the Enc processing 403 of the image 304 (B picture) is performed with reference to the locally decoded images of the images 303 and 306. Thereafter, Deb processing 404 of the image 304 is performed. In parallel to the Deb processing 404, Enc processing 405 of the image 305 (B picture) is performed with reference to the locally decoded image of the images 303 and 306. Thereafter, Deb processing 406 of the image 305 is performed. In parallel with the Deb processing 406, Enc processing 407 of the image 309 (P picture) is performed with reference to the locally decoded image of the image 306. Thereafter, Deb processing 408 of the image 309 is performed.

  As described above, the Deb processing 404 of the image 304 and the Enc processing 405 of the image 305, and the Deb processing 406 of the image 305 and the Enc processing 407 of the image 309 can be performed in parallel as pipeline processing. However, the Enc processing 403 of the image 304 cannot be performed in parallel with the Deb processing 402 of the image 303 because the local decoded image of the image 303 that has been encoded immediately before is used as a reference image. Therefore, in order to execute the Enc process 403, it is necessary to wait for the completion of the Deb process 402, and the process can not be performed in the pipeline.

  Next, with reference to FIG.5 and FIG.6, the encoding operation | movement in the case of restrict | limiting the search range which performs a motion vector detection is demonstrated. When encoding using the image encoded immediately before as a reference image, such as encoding of the image 304, the reference image range setting unit 118 sets a reference range (search range) for performing motion vector detection as usual. Also set a narrow range.

  FIG. 5 shows a reference image search range to which the image 304 refers. For encoding the image 304, the locally decoded image of the images 303, 306 is used as a reference image. FIG. 5 shows a local decoded image 501 of the image 303 and a local decoded image 502 of the image 306. A search range 503 indicated by shading is a search range of the local decoded image 501. Since the local decoded image 501 is not an image encoded immediately before the image 304, the search range 503 is the entire image. That is, the search range is not limited, and the search range is the same as in the normal coding operation. In the locally decoded image 502, an area 504 is a deblocking filtering portion of tile boundaries. Search ranges 505 and 506 (shaded portions) obtained by excluding the region 504 from the local decoded image 502 are search ranges of the local decoded image 502. Thus, when an image that has been encoded immediately before is used as a reference image, the reference image range setting unit 118 excludes the region 504 (boundary region of a plurality of tiles) from the search range. Therefore, the region 504 is not referred to when the image 304 is encoded, and the local decoded image 502 can be used as a reference image before performing the deblocking filtering process on the tile boundary (region 504).

  With reference to FIG. 6, the encoding operation of the images 306, 304, 305, and 309 when the search range is limited will be described. First, the Enc processing 601 of the image 306 (P picture) is performed with reference to the local decoded image of the image 303. Thereafter, Deb processing 602 of the image 306 is performed. In parallel to the Deb processing 602, Enc processing 603 of the image 304 (B picture) is performed with reference to the locally decoded images of the images 303 and 306. Thereafter, Deb processing 604 of the image 304 is performed. In parallel to the Deb processing 604, Enc processing 605 of the image 305 (B picture) is performed with reference to the locally decoded image of the images 303 and 306. Thereafter, Deb processing 606 of the image 305 is performed. In parallel with the Deb processing 606, the Enc processing 607 of the image 309 (P picture) is performed with reference to the local decoded image of the image 306. Thereafter, Deb processing 608 of the image 309 is performed.

  6 differs from the case where the search range is not limited (see FIG. 4) in that the Deb process 602 of the image 306 and the Enc process 603 of the image 304 are executed in parallel. Since Enc processing and Deb processing can be performed in parallel also when using a local decoded image encoded immediately before as a reference image, encoding processing can be performed while maintaining pipeline processing. Therefore, the processing time can be reduced.

  FIG. 7 is a flowchart of the encoding process executed by the imaging apparatus 100. Here, among the encoding processing, the processing mainly related to the limitation of the search range will be described, and the description of the other processing (for example, the encoded stream creation processing and the like) will be omitted. Also, the function of executing the processing of each step of this flowchart can be implemented by any hardware or software. For example, in the case of software implementation, these functions can be realized by the CPU of the imaging device 100 developing and executing a control program stored in the ROM in the RAM.

  In S701, the imaging apparatus 100 acquires an encoding target image. Acquisition of the encoding target image is performed according to a predetermined encoding order different from the input order of the images (see FIG. 3).

  In S702, the imaging apparatus 100 divides the encoding target image into a plurality of tiles.

  In step S <b> 703, the imaging apparatus 100 determines whether the encoding target image is an image encoded as an I picture. If the encoding target image is an image to be encoded as an I picture, the process proceeds to S704, and if not, the process proceeds to S705.

  In step S704, the imaging apparatus 100 performs encoding processing other than deblocking filtering processing of tile boundaries on the encoding target image in tile units (division area units). That is, the imaging apparatus 100 performs intra-frame encoding and decoding of each tile, and deblocking filter processing for a region excluding the tile boundary.

  If it is determined in S703 that the encoding target image is not an image to be encoded as an I picture, the encoding target image is an image to be encoded as a P picture or a B picture. That is, motion compensation interframe predictive encoding is performed on the encoding target image using one or more previously encoded images as reference images. With respect to the one or more reference images, the imaging device 100 determines in step S705 whether or not there is a reference image encoded immediately before. The “reference image encoded immediately before” is a reference image located immediately before the current encoding target image in the encoding order. If there is a reference image encoded immediately before, the process proceeds to S706, otherwise the process proceeds to S707.

  In S706, the imaging apparatus 100, using the reference image range setting unit 118, excludes tile boundaries from the search range of motion compensation interframe prediction for the reference image encoded immediately before.

  In step S <b> 707, the imaging apparatus 100 performs an encoding process other than the tile boundary deblocking filter process on the encoding target image in units of tiles. In other words, the imaging apparatus 100 performs motion compensation interframe predictive encoding and decoding of each tile, and deblocking filter processing on a region excluding the tile boundary. When the process of excluding the tile boundary from the search range is performed in S706, the search for the excluded tile boundary is not performed in the motion compensation interframe predictive coding of S707.

  In step S708, the imaging apparatus 100 performs tile boundary deblocking filter processing on the encoding target image that has been subjected to encoding processing other than the tile boundary deblocking filter processing in step S704 or S707. Note that the process of S708 can be executed in parallel with the process of S704 or S707 for the next encoding target image. Therefore, the imaging apparatus 100 executes the next process of S709 without waiting for the completion of the process of S708.

  In step S709, the imaging apparatus 100 determines whether the next encoding target image exists. If the next encoding target image exists, the process returns to S702. Otherwise, the encoding process of this flowchart ends. When the next encoding target image is encoded as a P picture or a B picture with reference to the current encoding target image, the current encoding is performed in S707 before the processing of S708 for the current encoding target image is completed. There is a possibility that the conversion target image is referred to. Even in this case, since tile boundaries are excluded from the search range in S705, the motion-compensated inter-frame prediction for the next encoding target image is performed in S707 before the processing of S708 on the current encoding target image is completed. It is possible to perform encoding.

  As described above, according to the first embodiment, the imaging apparatus 100 encodes and decodes the encoding target image in units of tiles. In addition, for an encoding target image corresponding to a P picture or a B picture, the imaging apparatus 100 performs encoding using motion compensation interframe prediction. An encoding target image used as a reference image for motion compensation inter-frame prediction of the current encoding target image may be positioned immediately before the current encoding target image in the encoding order. In this case, the imaging device 100 excludes the tile boundary area in the immediately preceding encoding target image from the search range of the motion compensation interframe prediction. This makes it possible to encode the current image to be encoded before the deblocking filter process on tile boundary areas in the immediately preceding image to be encoded is completed. Therefore, when filtering processing is performed on tile boundary areas in an image to be encoded in tile units, it is possible to suppress an increase in time required for encoding.

Second Embodiment
In the second embodiment, a configuration will be described in which motion compensation interframe prediction coding is performed without using an image coded immediately before as a reference image.

  FIG. 8 is a block diagram for explaining the arrangement of an imaging apparatus 800 including the image coding apparatus according to the second embodiment. The differences from the first embodiment will be mainly described below.

  The imaging apparatus 800 includes a reference image selection unit 801 instead of the reference image range setting unit 118 (see FIG. 1). The reference image selection unit 801 selects which image to use as a reference image for each picture. The reference relationship at the time of normal encoding will be described with reference to FIG. The image 306, which is a P picture, refers to the locally decoded image of the image 303. Images 304, 305, which are B pictures, refer to the locally decoded image of image 303 as a forward reference and to the locally decoded image of image 306 as a backward reference. As described above, in the B picture, two references, that is, a forward reference and a backward reference are usually performed.

  Here, the image 304 is located next to the image 306 in coding order. As described with reference to FIG. 4 in the first embodiment, when a local decoded image located immediately before in the encoding order is used as a reference image, it is necessary to wait for the Deb processing to end, and pipeline processing is performed. It can not be done. Therefore, the time required for encoding becomes long.

  In order to avoid this, in the present embodiment, the imaging apparatus 800 does not use the local decoded image located immediately before in the encoding order as a reference image. The reference image selection unit 801 selects a reference image to be used for encoding the current image to be encoded, based on the encoding order and the reference relationship at the time of normal encoding operation.

  With reference to FIG. 3 again, an example of reference image selection will be described. The image 306 uses a locally decoded image of the image 303 three pictures before in coding order as a reference image. The image 304 normally refers to the image 303 and the locally decoded image of the image 306. However, the image 306 is an image encoded immediately before. For this reason, the reference image selection unit 801 selects only a local decoded image of the image 303 encoded four pictures before as a reference image.

  Similar to the normal encoding operation, the image 305 uses, as reference images, a local decoded image of the image 303 encoded five pictures before and a local decoded image of the image 306 encoded two pictures before.

  When the image 304 is encoded, the image 306 positioned immediately before in the encoding order is not used as a reference image, so that the image 304 can be encoded without waiting for the Deb processing of the image 306. Processing can be performed.

  As described above, according to the second embodiment, the imaging apparatus 800 encodes and decodes the encoding target image in units of tiles. In addition, the imaging apparatus 800 performs encoding using motion compensation inter-frame prediction for an encoding target image corresponding to a P picture or a B picture. At this time, the imaging apparatus 800 does not use the encoding target image located immediately before the current encoding target image in the encoding order as a reference image for motion compensation inter-frame prediction. This makes it possible to encode the current image to be encoded before the deblocking filter process on tile boundary areas in the immediately preceding image to be encoded is completed. Therefore, when filtering processing is performed on tile boundary areas in an image to be encoded in tile units, it is possible to suppress an increase in time required for encoding.

Third Embodiment
FIG. 9 is a block diagram for explaining the arrangement of an imaging apparatus 900 including the image coding apparatus according to the third embodiment. The imaging device 900 includes an imaging unit 909, an image processing unit 901, an image encoding unit 902, a recording unit 903, a memory bus 904, a frame memory 905, a CPU 906, a flash memory 907, and a CPU bus 908.

  The imaging unit 909 includes a camera unit and an optical unit such as a lens and a CCD, and converts an optical signal captured from the lens into an electrical signal by a sensor. The imaging unit 909 outputs the converted electrical signal as Bayer-format digital RAW image data to a frame memory 905 such as a large-capacity DRAM via the memory bus 904.

The image processing unit 901 performs debayering processing (demosaicing processing) on the RAW image,
Convert to a signal consisting of luminance and color difference. After that, the image processing unit 901 performs so-called development processing such as removing noise contained in each signal, correcting optical distortion, and optimizing an image. Then, the image processing unit 901 outputs the image data after development to the frame memory 905 again in order to compress and encode the image data.

The image encoding unit 902 reads digital image data from the frame memory 905,
Video compression is performed using redundancy of image data by motion compensation interframe prediction and entropy coding, and an encoded bit stream is output. In the following description, the image encoding unit 902 adopts the HEVC encoding method as the video compression method, but the video compression method of the present embodiment is not limited to the HEVC encoding method.

  The recording unit 903 converts the encoded bit stream into a predetermined container format such as MP4 or MOV format that is compatible with reproduction and editing in various PC applications, and is used for a USB memory, an SD card, a hard disk, or the like. Record on a non-volatile memory medium.

  The memory bus 904 connects the imaging unit 909, the image processing unit 901, the image encoding unit 902, the recording unit 903, and the frame memory 905, and transfers image data, encoded data, and various parameter data at high speed. Data bus for The bus transfer method may be a standard bus standard such as ISA, PCI-Express, or AXI, or may be a unique bus method.

  The frame memory 905 stores original image data used in the encoding process of the image encoding unit 902, reference image data for inter-frame prediction, an encoded bit stream, and the like.

  The CPU 906 is a controller that performs hardware control such as activation and stop of the imaging unit 909, the image processing unit 901, the image encoding unit 902, and the recording unit 903, and interrupt notification via the CPU bus 908.

  The flash memory 907 is a nonvolatile memory that stores a program for operating the CPU 906. The CPU 906 can access the flash memory 907 by a fetch operation.

  The CPU bus 908 is a control bus that connects the CPU 906 and each peripheral. The CPU bus 908 may adopt a standardized bus standard method similar to the memory bus 904, or may adopt a low-speed serial method such as I2C if there is sufficient processing margin.

  Next, details of the image coding unit 902 will be described with reference to FIG. The image coding unit 902 includes an intra prediction unit 1000, an inter prediction unit 1001, an intra / inter determination unit 1002, a predicted image generation unit 1003, an integer conversion unit 1004, a quantization unit 1005, an entropy coding unit 1006, and a code amount control. And a unit 1007. The image coding unit 902 also includes an inverse quantization unit 1008, an inverse integer conversion unit 1009, a loop filter 1010, a post loop filter 1011, a subtractor 1012, and an adder 1013.

  The intra prediction unit 1000 reads the image data of the encoding target block from the frame image data stored in the frame memory 905, and calculates correlations with a plurality of intra prediction images generated from the reference pixel data around the encoding target block. Do. Then, the intra prediction unit 1000 selects the intra prediction method with the highest correlation and notifies the intra / inter determination unit 1002 of the intra prediction method.

  The inter prediction unit 1001 receives the encoding target image stored in the frame memory 905, and receives encoded image data as a reference image. Then, the inter prediction unit 1001 performs pattern matching of pixel data on a block basis to calculate a motion vector.

  The intra / inter determination unit 1002 selects a prediction scheme for encoding based on the results of the output from the above-described intra prediction unit 1000 and inter prediction unit 1001. As a specific selection method, for the prediction image calculated by the intra prediction unit 1000 for the encoding target image block and the prediction image generated from the reference image using the motion vector derived by the inter prediction unit 1001, the encoding target The prediction errors with the image are respectively derived and compared. Alternatively, a method may be used in which a prediction error is obtained by the intra prediction unit 1000 or the inter prediction unit 1001, and after the prediction error is obtained as an evaluation value by the intra / inter determination unit 1002, the evaluation values are compared. As a result of comparison of prediction errors, the intra / inter determination unit 1002 determines the prediction mode with the smaller difference value as the coding prediction mode, and outputs this to the prediction image generation unit 1003.

  The predicted image generation unit 1003 generates a predicted image according to the prediction mode selected by the intra / inter determination unit 1002. The generated predicted image is output to the subtractor 1012 for calculating a difference image with the input image. The predicted image is also output to the adder 1013 for local decoded image generation.

  The integer conversion unit 1004 performs a spatial resolution conversion of the pixel data in units of blocks and converts it into a spatial frequency domain.

  The quantization unit 1005 calculates a quantization coefficient based on the target code amount, and performs quantization processing on the coefficient data converted into the spatial frequency domain by the integer conversion unit 1004. The quantized coefficient data (quantized data) is output to the entropy encoding unit 1006 for entropy encoding. The quantized coefficient data is also output to the inverse quantization unit 1008 to calculate a reference image or a predicted image.

  The entropy coding unit 1006 compresses information by entropy coding using coefficient generation probability of bit data such as CABAC (context adaptive arithmetic coding) method for coefficient data quantized by the quantization unit 1005. I do. In the case of inter prediction, the entropy encoding unit 1006 also performs entropy encoding on the vector value used in motion prediction. The entropy encoding unit 1006 adds parameters (header information such as SPS and PPS) necessary for the decoding process to the encoded data, shapes the encoded data into a predetermined data format, and outputs the data to the frame memory 905. The output data is stored in the frame memory 905, converted into a predetermined container format by the recording unit 903, and then saved as a file on a recording medium.

  The code amount control unit 1007 acquires the code amount of the encoded data output from the entropy encoding unit 1006, and calculates the target code amount per picture based on the bit rate and the buffer model. Then, the code amount control unit 1007 performs feedback control to set the calculated target code amount in the quantization unit 1005.

  The inverse quantization unit 1008 calculates coefficient data by multiplying the coefficient data quantized by the quantization unit 1005 by the quantization coefficient again.

  The inverse integer transform unit 1009 generates image data by performing inverse integer transform on the coefficient data output from the inverse quantization unit 1008.

  The loop filter 1010 performs a filter process (for example, a deblocking filter process) that reduces coding distortion generated at the block boundary on the image data obtained by adding the image data output from the inverse integer transform unit 1009 and the predicted image. Apply. The loop filter 1010 outputs the filtered image data to the frame memory 905 as a locally decoded image. When the image to be encoded is tile-divided and encoded in the horizontal direction defined by the HEVC standard, the loop filter 1010 skips the filter processing for pixels that need to be filtered across tile boundaries.

  The post-loop filter 1011 operates independently of the tile-by-tile encoding process. The post-loop filter 1011 reads again the pixel data of the tile boundary (the boundary area of the divided area) in which the filter processing is skipped by the loop filter 1010 with respect to the local decoded image for one screen stored in the frame memory 905 Do the processing.

  Note that the entropy encoding unit 1006 and the code amount control unit 1007 described above normally perform control in units of pictures for one screen, and the other processing blocks are controlled in units of predetermined rectangular pixel blocks. In particular, in the HEVC scheme, quantization is performed in units of CU (Coding Unit), motion vector search is performed in PU (Prediction Unit), and integer conversion is performed in units of TU (Transform Unit).

  FIG. 11 is a flowchart of the encoding process executed by the imaging apparatus 900. Here, among the encoding processing, processing mainly related to control of the encoding order will be described, and description of the other processing (for example, encoding stream creation processing and the like) will be omitted. When the imaging apparatus 900 enters a moving image recording state triggered by a user operation from an interface (not shown) such as a touch panel or a button, the processing of this flowchart is executed every frame period.

  First, in S1101, the image encoding unit 902 determines whether or not the imaging apparatus 900 is recording a moving image. When a moving image is being recorded, the encoding process for one screen described with reference to FIG. 10 starts, and the process of this flowchart proceeds to S1102. If a moving image is not being recorded, the image encoding unit 902 determines that a moving image recording stop instruction has been issued by a user operation, and ends the processing of this flowchart.

  In step S1102, the image encoding unit 902 determines whether the picture type of the image to be encoded is an I picture or a P picture used as a reference picture. If the encoding target image is an I picture or a P picture, the processing proceeds to step S1103. If not (ie, if the encoding target image is a B picture not used as a reference picture), the processing proceeds to step S1106.

  In step S1103, the image encoding unit 902 performs screen division using tiles on the encoding target image, and executes image encoding for one screen in one frame period in units of tiles.

  In step S1104, the image encoding unit 902 determines whether encoding of all tiles, that is, one entire picture has been completed. This determination is performed at the timing when the encoding process in units of tiles is completed. The method for detecting the tile-by-tile encoding completion is not particularly limited. For example, a method of repeatedly inquiring the operation state by polling, a method of interrupt notification, or the like can be used. If all the tiles have been encoded, the process proceeds to S1105; otherwise, the process returns to S1103.

  In step S <b> 1105, the image encoding unit 902 performs filter processing (hereinafter also referred to as “post LF processing”) on the tile boundary of the local decoded image of the encoding target image using the post-loop filter 1011. The post-loop filter 1011 operates independently of the other blocks in the image encoding unit 902. Therefore, the image encoding unit 902 can return the processing to S1101 and execute encoding of the subsequent picture without waiting for completion of the post-LF processing.

  In step S1106, the image encoding unit 902 determines whether or not the previously encoded I or P picture is used as a reference image of the current encoding target B picture. If the previously encoded I or P picture is used as a reference image for the current encoding target B picture, the process proceeds to S1107; otherwise, the process proceeds to S1108.

  When the previously encoded I or P picture is used as a reference image of the current encoding target B picture, there is a possibility that the post LF processing for the tile boundary of the reference image has not yet been completed. Therefore, if the current encoding target B picture is encoded at this timing, the tile boundary will be referred to the locally decoded image which is not filtered. Therefore, in step S1107, the image encoding unit 902 is configured to encode the I or P picture (the I or P picture captured next to the I or P picture encoded last time) subsequent in the input order first. Sort the order of transformation. This makes it possible to start coding of the B picture after the post LF processing on the picture used as the reference image is completed. In order to enable rearrangement of the encoding order, the frame memory 905 is provided with a buffer having a sufficient capacity that allows a processing delay due to rearrangement.

  When the determination in S1106 is performed again for the B picture whose coding order is delayed due to the above rearrangement, the previously coded I or P picture is not used as a reference image of the current B object picture to be coded. Therefore, the process proceeds to S1108. In this case, post LF processing of an I or P picture used as a reference image is considered to be already completed. Therefore, in step S1108, the image encoding unit 902 encodes a B picture.

  Next, an example of the encoding operation according to the flowchart of FIG. 11 will be described with reference to FIG. Here, a case will be described where a B picture refers to an I picture or a P picture captured before and after. 12 shows the input order of the encoding target images (the order acquired by the imaging unit 909, developed by the image processing unit 901, and input to the frame memory 905), and the encoding order (image encoding units other than post-LF processing). The order of the encoding process 902) is shown. In addition, the execution order of post LF processing is also shown. FIG. 12 shows the picture type and the input order of each picture as a subscript of the picture type. Further, with regard to the B picture, the pictures to be referred to are associated by the arrows. In the example of FIG. 12, it is assumed that encoding with tile division of right and left is performed, but the configuration of tile division is not limited to this.

  The imaging apparatus 900 has hardware performance such that it is completed within a predetermined frame period, such as 30 Hz or 60 Hz, for any of input, encoding, and post-LF processing.

  After the encoding of the I0 picture and the P3 picture is completed according to the flowchart of FIG. 11, the encoding of the B1 picture starts with the conventional method. However, in the present embodiment, the P6 picture is encoded before the B1 picture. In other words, before coding the B1 picture, all the pictures (I0 picture and P3 picture) used as a reference picture at the time of coding the B1 picture among the I and P pictures, and the picture photographed next (P6 picture) is encoded. Therefore, at the timing when the B1 picture is encoded, the post LF processing of the P3 picture referred to by the B1 picture is completed. As a result, when the B1 picture is encoded, it is possible to refer to the I0 picture and the P3 picture that have been subjected to the filtering process on the tile boundary. The same effect can be obtained for B pictures after B2 picture. In addition, since encoding can be performed in all frame periods after the P6 picture, an increase in time required for encoding can be suppressed.

  As described above, according to the third embodiment, the imaging apparatus 900 encodes and decodes the encoding target image in tile units. In addition, for an image to be encoded corresponding to a P picture or a B picture, the imaging device 900 performs encoding using motion compensated inter-frame prediction. Prior to encoding the B picture, the imaging apparatus 900 encodes all I or P pictures used as reference images at the time of encoding the B picture and the next captured I or P picture. . This makes it possible to suppress an increase in time required for encoding when filtering processing is performed on tile border areas in an image encoded in tile units.

Fourth Embodiment
In the third embodiment, control of the encoding order has been described. In the fourth embodiment, a configuration for controlling the order of entropy encoding performed by the entropy encoding unit 1006 will be described. In the fourth embodiment, the basic configuration of the imaging apparatus 900 is the same as that of the third embodiment (see FIGS. 9 and 10). The differences from the third embodiment will mainly be described below.

  FIG. 13 is a flowchart of the entropy coding process performed by the imaging apparatus 900. The process of this flowchart is executed in parallel with the encoding process according to the flowchart of FIG. Moreover, although the process of this flowchart shall be started for every frame period, a starting timing in particular is not limited. For example, the CPU 906 may detect the end timing of image encoding processing in units of one frame, the end timing of post-LF processing, and the like by an interrupt notification, and the processing of this flowchart may be started using a completion interrupt as a trigger.

  In step S1301, the entropy coding unit 1006 determines whether or not unprocessed quantized data not yet subjected to entropy coding remains in the frame memory 905. If unprocessed quantized data remains in the frame memory 905, the process proceeds to S1302. If unprocessed quantized data does not remain in the frame memory 905, the processing of this flowchart ends, and is activated again at the next frame cycle timing.

  In step S1302, the entropy encoding unit 1006 determines whether the picture type of the first quantized data (current quantized data) waiting for processing is an I picture or a P picture. If the picture type is I picture or P picture, the process proceeds to S1303; otherwise, the process proceeds to S1307.

  In step S1303, the entropy coding unit 1006 determines whether entropy coding has already been performed on the quantization data of the reference number of pictures. For example, in the case of a GOP structure called M = 3 structure, a case where a B picture refers to a total of two pictures, one before and after, is common. In this case, in S1303, the entropy encoding unit 1006 checks whether or not the entropy encoding of the I or P picture of 2 pictures has been completed. In order to make this determination, the entropy coding unit 1006 has information that the reference number is 2, and has a counting mechanism that counts the number of reference pictures VeRefCnt that have been entropy coded. Then, the entropy coding unit 1006 determines whether the counter value VeRefCnt has become 2 or not. If the quantization data of the reference number of pictures has not been entropy coded, the process proceeds to S1304, and if the quantization data of the reference number of pictures has been entropy coded, the process proceeds to S1306.

  In step S1304, the entropy coding unit 1006 reads out the current quantized data from the frame memory 905, performs entry-piece coding, and outputs it as a bit stream.

  In S1305, the entropy coding unit 1006 increments the counter value VeRefCnt. Thereafter, the process returns to S1301.

  In S1306, the entropy coding unit 1006 rearranges the current quantization data behind a queue that manages the order of entropy coding, in order to entropy code the subsequent B pictures first.

  In S1307, the entropy encoding unit 1006 entropy-encodes the current quantized data (in this case, corresponding to a B picture), and outputs it as a bit stream. Here, S1307 and S1304 have the same processing contents except for the picture type of the quantized data to be processed.

  In step S1308, the entropy encoding unit 1006 determines whether entropy encoding of the B picture has been performed for the number of consecutive B pictures between two I or P pictures. Here, a data structure is assumed in which two B pictures are successively inserted between two I or P pictures. If the entropy coding of two consecutive B pictures is completed, the processing proceeds to S1309, and if not, the processing returns to S1301.

  In S1309, the entropy coding unit 1006 decrements the counter value VeRefCnt. This is because in this case, the entropy coding is not performed on the forward picture among the two pictures referred to by the next B picture.

  Next, an example of the entropy coding operation according to the flowchart of FIG. 13 will be described with reference to FIG. FIG. 14 shows the input order of the encoding target image, the encoding order, and the execution order of the post LF processing, as in FIG. The order of these is the same as in FIG. In addition to this, FIG. 14 shows the order of entropy coding.

  When coding (except entropy coding) of one image is completed, the frame memory 905 stores the quantized data output from the quantization unit 1005 in addition to the locally decoded image for one picture. According to the flowchart of FIG. 13, since the counter value VeRefCnt is less than 2 at the timing at which the first two pictures of the I0 picture and the P3 picture are encoded, the I0 picture and the P3 picture are entropy-encoded in succession.

  Subsequently, although encoding of P6 is completed in the encoding order, entropy encoding of reference pictures for two pictures has already been completed. Therefore, the order of the entropy coding in S1306 of FIG. 13 is rearranged, and the entropy coding of the quantized data of the P6 picture is postponed. As a result, the entropy coding is continuously performed for the B1 picture and the B2 picture, and the counter value VeRefCnt is decremented to one. Thereafter, entropy coding of the P6 picture is performed. The order rearrangement is similarly performed for the P9 picture and subsequent pictures, and the P picture not referenced by the B picture is entropy coded later than the B picture.

  In this way, by reordering the entropy coding order and reordering the stream output, three reference pictures of the reference picture are decoded and buffered, and then the B picture is decoded to perform troublesome display order reordering. Less need to be done. As a result, the output delay can be shortened, and the buffer on the frame memory 905 can be reduced.

  As described above, according to the fourth embodiment, the image capturing apparatus 900 uses the front of an I or P picture photographed next to all I or P pictures used as reference images when encoding a B picture. Then, entropy coding of this B picture is performed. As a result, the output delay can be shortened and the required amount of frame memory can be reduced.

[Other Embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 105 ... Motion estimation part, 106 ... Orthogonal transformation part, 107 ... Quantization part, 114 ... Deblocking filter part, 116 ... Reference frame buffer, 117 ... Tile boundary deblocking filter part

Claims (14)

An image coding apparatus that codes a plurality of images included in a moving image according to a predetermined coding order,
A division unit configured to divide a first image included in the moving image into a plurality of divided areas;
Encoding means for encoding the first image in units of divided areas, and encoding a second image included in the moving image using motion compensated inter-frame prediction;
Decoding means for decoding the encoded first image;
Filter means for performing filter processing on boundary regions of the plurality of divided regions in the decoded first image;
With
If the first image is located immediately before the second image in the predetermined encoding order and the first image is used as a reference image for the motion compensated inter-frame prediction, the encoding An image coding apparatus, wherein the means excludes the boundary area of the plurality of divided areas in the first image from the search range of the motion compensation interframe prediction. An image coding apparatus that codes a plurality of images included in a moving image according to a predetermined coding order,
A division unit configured to divide a first image included in the moving image into a plurality of divided areas;
Encoding means for encoding the first image in units of divided areas, and encoding a second image included in the moving image using motion compensated inter-frame prediction;
Decoding means for decoding the encoded first image;
Filter means for performing filter processing on boundary regions of the plurality of divided regions in the decoded first image;
With
When the first image is located immediately before the second image in the predetermined encoding order, the encoding means uses the first image as a reference image for the motion compensation inter-frame prediction. An image coding apparatus characterized in that: When the first image is located immediately before the second image in the predetermined encoding order, the filtering process for the first image by the filter means and the second by the encoding means The image coding apparatus according to claim 1, wherein the image coding is performed in parallel. The image coding apparatus according to any one of claims 1 to 3, wherein the filtering process includes a deblocking filtering process. The image coding apparatus according to any one of claims 1 to 4, wherein the plurality of divided areas are tiles of HEVC coding scheme. An image code that encodes a moving image including a first image that is not used as a reference image for motion compensation inter-frame prediction and a plurality of second images that are used as reference images for motion compensation inter-frame prediction Device.
Division means for dividing each of the plurality of second images into a plurality of divided areas;
Encoding means for encoding each of the plurality of second images in units of divided areas, and encoding the first image using motion compensated inter-frame prediction;
Decoding means for decoding each of the plurality of encoded second images;
Filter means for performing filter processing on boundary regions of the plurality of divided regions in each of the plurality of decoded second images;
With
The encoding means, before encoding the first image, out of the plurality of second images, all images used as reference images when encoding the first image and all the images An image coding apparatus characterized by coding an image and an image captured next to the image. Entropy encoding means for entropy encoding each of the encoded first image and the plurality of second images,
The image encoding apparatus according to claim 6, wherein the entropy encoding unit performs entropy encoding on the first image before entropy encoding the next photographed image. The image coding apparatus according to claim 6, wherein the filtering process includes a deblocking filtering process. The image coding apparatus according to any one of claims 6 to 8, wherein the plurality of divided regions are tiles of HEVC coding scheme. The image coding apparatus according to any one of claims 1 to 9.
Imaging means for generating the moving image;
An imaging apparatus comprising: An image encoding method executed by an image encoding device that encodes a plurality of images included in a moving image according to a predetermined encoding order,
A dividing step of dividing a first image included in the moving image into a plurality of divided regions;
Encoding the first image in units of divided regions, and encoding a second image included in the moving image using motion compensated inter-frame prediction;
A decoding step of decoding the encoded first image;
Filtering the boundary region of the plurality of divided regions in the decoded first image;
With
If the first image is located immediately before the second image in the predetermined encoding order and the first image is used as a reference image for the motion compensated inter-frame prediction, the encoding An image coding method characterized in that the step excludes the boundary area of the plurality of divided areas in the first image from the search range of the motion compensation interframe prediction. An image encoding method executed by an image encoding device that encodes a plurality of images included in a moving image according to a predetermined encoding order,
A dividing step of dividing a first image included in the moving image into a plurality of divided regions;
Encoding the first image in units of divided regions, and encoding a second image included in the moving image using motion compensated inter-frame prediction;
A decoding step of decoding the encoded first image;
Filtering the boundary region of the plurality of divided regions in the decoded first image;
With
When the first image is located immediately before the second image in the predetermined encoding order, the encoding step uses the first image as a reference image for the motion compensation interframe prediction. An image coding method characterized in that Image code for encoding a moving image including a first image not used as a reference image for motion compensated interframe prediction and a plurality of second images used as a reference image for motion compensated interframe prediction An image encoding method executed by the encoding device,
A dividing step of dividing each of the plurality of second images into a plurality of divided regions;
Encoding each of the plurality of second images in units of divided regions, and encoding the first image using motion compensated interframe prediction;
A decoding step of decoding each of the plurality of encoded second images;
A filter step of performing filter processing on boundary regions of the plurality of divided regions in each of the plurality of decoded second images;
With
In the encoding step, before encoding the first image, all the images used as a reference image in encoding the first image among the plurality of second images and all the images. An image encoding method comprising encoding an image and an image captured next to the image.   A program for causing a computer to function as each means of the image coding device according to any one of claims 1 to 9.

Images