JP5682387B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for processing a moving image.

  Due to the widespread use of high-definition broadcasting and the like, images handled by the moving image processing circuit are becoming larger. As the screen size and functionality increase, the external memory bandwidth and the external memory size used by the moving image processing circuit are steadily increasing.

  In general, in a moving image processing system LSI (Large Scale Integration), a single image processing circuit is not connected, and a plurality of circuits such as an image enlargement / reduction circuit and a filter processing circuit are connected in multiple stages. Processing circuits connected in multiple stages are subjected to pipeline processing (sequential processing), for example, in units of pictures.

  FIG. 1 is a diagram for explaining pipeline processing in units of pictures. In the example shown in FIG. 1, the first image processing circuit 101, the second image processing circuit 102, and the third image processing circuit 103 are pipeline processed to obtain desired moving image data.

  The external memory 105 stores an input picture 111, a picture 112 processed by the first image processing circuit 101, a picture 113 processed by the second image processing circuit 102, and a picture 114 processed by the third image processing circuit 103. Is done.

  In the pipeline processing as shown in FIG. 1, when a new image processing circuit is connected, a memory area that secures a memory bandwidth by the amount of the image processing circuit reading and writing a picture from the external memory 105 via the memory bus 104. And will increase.

  For example, in the example shown in FIG. 1, the size of one picture is about 3 MB, and the frame rate of the system LSI is 30 fps. For example, each image processing circuit reads an input image and writes a processed image 30 times per second, so that the total data transfer amount is 1 picture 3 MB × 2 times read / write × 30 times × 3. Block = 540 MB / s≈4.2 Gbps.

  Thus, when a plurality of moving image processing modules are connected, the memory bandwidth increases in proportion to the number of moving image processing modules. Therefore, there is known a technique for reducing the memory bandwidth by compressing an image output when the image processing circuit writes a picture in the external memory and expanding and processing the compressed data when reading the picture.

  In this technique, since the picture data written to the external memory is compressed, it is possible to reduce the memory bandwidth corresponding to the compression. In this technique, when a picture is reversibly compressed, the effect of reducing the external memory bandwidth cannot be obtained qualitatively, and in the reversible compression process, the theoretically worst compressed data size is larger than that of an uncompressed image.

  Therefore, when performing lossless compression, there is no effect in reducing the external memory size. Therefore, in order to reduce the external memory bandwidth and the external memory size, it is necessary to perform irreversible compression on the image output from each image processing circuit.

  However, even if a general irreversible compression algorithm such as JPEG (Joint Photographic Experts Group) is adopted, it is not always possible to qualitatively reduce the external memory size and memory bandwidth.

  This is because, in the lossy compression algorithm, the amount of image information is reduced by quantization processing, and therefore the size of the data size after lossy compression differs depending on the image characteristics of the compression target picture. Therefore, it is difficult to appropriately estimate the data size after compression even if an irreversible compression algorithm is used.

  In general, in order to solve such a problem, there is a process called a 2-pass process. “Two-pass processing” means that the image to be compressed is scanned once before compression, the feature amount is calculated, and the irreversible compression processing is actually performed based on the result. This is a process for generating compressed data that is very close to the target data size.

  FIG. 2 is a diagram illustrating an example of 2-pass processing. In the moving image encoding apparatus shown in FIG. 2, the target image is stored in a delay circuit (FIFO) 201, and the encoding process of the target image is delayed by the time corresponding to the encoding of one image. While the encoding process is delayed, the approximate code amount is estimated by the simple statistical information extraction unit 202. The encoding processing unit 203 encodes the encoding target image stored in the delay circuit 201 using the correction coefficient obtained from the statistic extraction unit 202.

JP 2010-74705 A Japanese Patent Laid-Open No. 10-66067 JP 2006-246276 A JP 10-42293 A

  However, in the prior art, it has been impossible to appropriately reduce the size of the memory for storing the image while efficiently processing the image in the pipeline processing. In the two-pass process that realizes the qualitative reduction of the image data size, the screen must be scanned once as described above, which is not an efficient process.

  Accordingly, the disclosed technology has been made in view of the above problems, and an image processing apparatus capable of appropriately reducing the size of a memory for storing an image while efficiently processing the image in pipeline processing, and An object is to provide an image processing method.

An image processing apparatus according to an aspect of the disclosure is an image processing apparatus that performs pipeline processing on processing of a first image processing unit and a second image processing unit, and compresses a current image output from the first image processing unit. A first compression unit stored in the storage unit, a decompression unit that decompresses the data compressed by the first compression unit read from the storage unit and supplies the decompressed data to the second image processing unit, and the second image processing A second compression unit for compressing an image output from the first unit, a data size of the current image compressed by the first compression unit, and a data size of the past image compressed by the first and second compression units, respectively. A control unit that estimates a data size of the current image after being compressed by the second compression unit using a ratio, and controls compression of the current image by the second compression unit based on the estimated data size; .

  According to the disclosed technology, the size of a memory for storing an image can be appropriately reduced while efficiently processing the image in pipeline processing.

The figure for demonstrating the pipeline process of a picture unit. The figure which shows an example of 2 path | pass processing. The block diagram which shows an example of a structure of the image processing apparatus in a comparative example. 1 is a block diagram illustrating an example of a configuration of an image processing apparatus according to an embodiment. The block diagram which shows an example of a structure of a 1st compression part. The block diagram which shows an example of a structure of a 2nd compression part. The figure for demonstrating a size table. The figure for demonstrating reversible / irreversible selection information. The figure which shows the example of the compression result showing whether each block of the past image was lossless compression or lossy compression. The figure for demonstrating correction | amendment of an estimated value. The flowchart which shows an example of the process of a past image. 10 is a flowchart illustrating an example of a current image estimation process. The flowchart which shows an example of the production | generation process of reversible / irreversible selection information. The flowchart which shows an example of a reversible / irreversible selection process. The flowchart which shows another example of the reversible / irreversible selection process.

  First, a comparative example to be compared with the embodiment will be described. Although the two-pass processing as in the conventional technique can reduce the qualitative data size of an image, the screen must be scanned once to perform the two-pass processing as described above.

  In addition, images output from the image processing circuit are not written to the external memory at a time, but are generally sequentially output in raster scan order or small area (block) units of the image. Therefore, in order to apply 2-pass processing that requires scanning the entire screen before compressing the image output by the image processing circuit in units of blocks, a large-capacity storage device that can store at least one screen is required. It becomes.

  When the storage area is obtained from the external storage device, the transfer capacity of the external memory further increases. Therefore, when compressing the output image of the preceding moving image processing circuit, it is conceivable to calculate statistical information and use the statistical information for compressing the output image of the subsequent moving image processing circuit. This process will be described below as a comparative example.

[Comparative example]
FIG. 3 is a block diagram illustrating an example of the configuration of the image processing apparatus 30 in the comparative example. 3 includes a first image processing unit 301, a first compression unit 302, an estimation unit 303, a memory 304, a first expansion unit 305, a second image processing unit 306, and a second compression unit 307. . Each process in the image processing apparatus 30 is pipelined.

  The first image processing unit 301 performs image processing on the processing target image. The processed image data is output to the first compression unit 302.

  The first compression unit 302 performs compression processing on the image data acquired from the first image processing unit 301. At this time, the statistical information after the compression processing is output to the estimation unit 303. The statistical information is, for example, the data size before and after compression. The compressed image data is stored in the memory 304.

  The estimation unit 303 estimates the data size after compression by the second compression unit 307 using the statistical information acquired from the first compression unit 302. The estimated value estimated is output to the second compression unit 307. Thereby, it is not necessary to scan the same image twice, and the pipeline processing can be performed efficiently.

  The memory 304 is, for example, a DRAM (Dynamic Random Access Memory), and stores image data output from each processing unit.

  The first expansion unit 305 expands the image data compressed by the first compression unit 302 and outputs the expanded image to the second image processing unit 306.

  The second image processing unit 306 performs image processing on the image acquired from the first development unit 305. The processed image data is output to the second compression unit 307.

  The second compression unit 307 compresses the image data acquired from the second image processing unit 306 based on the estimated value acquired from the estimation unit 303. As a result, the size after compression can be adjusted based on the estimated value. Therefore, it is sufficient to prepare the capacity of the memory 304 corresponding to this size, and the size of the memory 304 can be reduced.

  On the other hand, the image characteristics of each image may change between the image after image processing by the second image processing unit 306 and the image after image processing by the first image processing unit 301. In this case, it is not appropriate to apply the statistical information of the first compression unit 302 to the second compression unit 307.

  Here, in irreversible compression, compression efficiency is improved by deleting high frequency components of an image. However, when the second image processing unit 306 includes low-pass filter processing or unsharp mask processing, the characteristics of the high-frequency component of the image processed by the second image processing unit 306 change.

  Therefore, the characteristics of the compression target image in the first compression unit 302 and the characteristics of the compression target image in the second compression unit 307 may be completely different. In this case, even if the data size after compression of the second compression unit 307 is estimated simply using the statistical information at the time of compression by the first compression unit 302, the estimated data size and the actually compressed data size are: It can be quite different.

  Therefore, when the configuration of the image processing apparatus 30 shown in FIG. 3 is considered, it is only necessary that the data size after compression by the second compression unit 307 can be estimated appropriately. Hereinafter, an embodiment in which the data size after the subsequent compression processing can be appropriately estimated will be described.

[Example]
FIG. 4 is a block diagram illustrating an example of the configuration of the image processing apparatus 40 in the embodiment. 4 includes a memory 400, a first image processing unit 401, a first compression unit 402, a first expansion unit 403, a second image processing unit 404, a second compression unit 405, and a second expansion unit 406. And an output unit 407 and a control unit 408. The image processing apparatus 40 can be realized as a system LSI.

  The memory 400 stores data input / output to / from each processing unit of the image processing apparatus 40. The memory 400 is a DRAM, for example, and may be provided as an external memory different from the image processing apparatus 40.

  The first image processing unit 401 performs image processing on the processing target image input from the memory 400. The first image processing unit 401 is, for example, an MPEG2 decoding device that performs decoding by the MPEG (Moving Picture Expert Group) 2 method.

  The image data processed by the first image processing unit 401 is output to the first compression unit 402. The first image processing unit 401 performs image processing for each of a plurality of blocks obtained by dividing an image. This block is, for example, a 16 × 16 macro block. In this case, the first image processing unit 401 outputs the block unit to the first compression unit 402.

  The first compression unit 402 performs compression processing on the image data acquired from the first image processing unit 401. For example, the first compression unit 402 performs lossless compression processing independently for each block. In the lossless compression process, for example, a variable length encoding process such as a Huffman code is performed on the difference data from adjacent pixels. The first compression unit 402 does not necessarily perform lossless compression, and may perform lossy compression.

  FIG. 5 is a block diagram illustrating an example of the configuration of the first compression unit 402. The first compression unit 402 illustrated in FIG. 5 includes a first encoding device 421. The first encoding device 421 acquires a block from the first image processing unit 401 and performs an encoding process for each block. The encoding process is a reversible variable length encoding process as described above.

  The first encoding device 421 stores the encoded first encoded data in the memory 400. Further, the first encoding device 421 stores the data of the block that has been subjected to the encoding process and has become a variable length in the memory 400 as first size information (for example, a first size table). As a result, the first size table can be randomly accessed.

  Returning to FIG. 4, the first expansion unit 403 acquires the first encoded data stored in the memory 400 and performs expansion processing. The decompression process is a variable length decoding process corresponding to the variable length encoding process by the first compression unit 402.

  The first expansion unit 403 refers to the first size table stored in the memory 400, calculates an address where desired block data is stored, reads out the desired first encoded data, and performs expansion processing. You may go. The developed image data is output to the second image processing unit 404.

  The second image processing unit 404 performs image processing on the image data acquired from the first development unit 403. The second image processing unit 404 is, for example, a deblocking filter device for removing block noise unique to the MPEG2 decoded image. The second image processing unit 404 outputs the filtered image data to the second compression unit 405.

  The second compression unit 405 performs reversible compression or irreversible compression on the image data acquired from the second image processing unit 404 based on selection information indicating reversible or irreversible for each block generated by the control unit 408. Do. The lossless compression process may be the same as the process performed by the first compression unit 402, and the lossy compression process may be a quantization process, a bit rounding process, or the like.

  This selection information is also referred to as reversible / irreversible selection information. The lossless / irreversible selection information includes the ratio of the data size after compression by the second compression unit 405 to the data size after compression by the first compression unit 402 for the past image and the compression of the current image by the first compression unit 402. It is generated based on the data size. The reversible / irreversible selection information is also control information for controlling the compression of the second compression unit 405. Details of the reversible / irreversible selection information will be described later.

  FIG. 6 is a block diagram illustrating an example of the configuration of the second compression unit 405. The second compression unit 405 illustrated in FIG. 6 includes a second encoding device 451. The second encoding device 451 performs lossless compression processing or lossy compression processing on the block acquired from the second image processing unit 404 based on the lossless / lossy selection information.

  The second encoding device 451 stores the encoded second encoded data in the memory 400. The second encoding device 451 stores the encoded data size of each block in the memory 400 as second size information (for example, a second size table). Thereby, the second size table can be randomly accessed.

  Returning to FIG. 4, the second expansion unit 406 acquires the second encoded data stored in the memory 400 and performs expansion processing. The decompression process is a decoding process corresponding to the encoding process performed by the second compression unit 406. The developed image data is output to the output unit 407.

  The output unit 407 outputs the image data acquired from the second development unit 406. For example, when the output unit 407 is a display device, the output unit 407 displays image data. The output unit 407 may be a device that outputs image data to a subsequent processing unit (for example, a storage unit).

  The control unit 408 is, for example, a processor, and controls each unit or estimates the data size after compression by the second compression unit 405. For example, the control unit 408 includes an estimation unit 409.

  The estimation unit 409 performs estimation processing for estimating the data size after compression by the second compression unit 405 as described above. The estimation unit 409 calculates the ratio of the data size after compression of the second compression unit 405 to the data size after compression of the first compression unit 402 using the first size table and the second size table for the past image. .

  A plurality of past images may be used as the past image, but the image immediately before the current image may be used based on the idea that the immediately preceding image is similar to the current image in consideration of the correlation of the images. .

  The estimation unit 409 multiplies the calculated ratio by each data in the first size table of the current image, and estimates the data size (second size table) of the current image after being compressed by the second compression unit 405.

  The estimation unit 409 compares the estimated total value of each data in the second size table with a threshold value indicating the target data size. If the total value is less than or equal to the threshold, the estimation unit 409 generates reversible / irreversible selection information in which reversibility is set for all blocks, and stores the reversible / irreversible selection information in the memory 400.

  If the total value is larger than the threshold, the estimating unit 409 sets one or a plurality of predetermined blocks to irreversible compression based on the estimated second size table, generates lossless / irreversible selection information, and stores the memory 400. The predetermined blocks may be all blocks or may be determined in descending order of the estimated size.

  Thereby, it is possible to determine reversible or irreversible in units of blocks, and it is possible to maintain good image quality by reducing blocks that are irreversibly compressed.

  Note that the estimation unit 409 may calculate the estimation error by comparing the estimated second size table with the second size table actually compressed by the second compression unit 405 for the past image. . The estimation unit 409 corrects the second size table estimated for the current image using the estimation error.

  The estimation unit 409 may generate lossless / irreversible selection information using the corrected second size table. Thereby, the estimation accuracy can be improved.

<Data example>
Next, a data example of each unit will be described. First, an example of the size table will be described. FIG. 7 is a diagram for explaining the size table. Both the first size table and the second size table have the same data structure as the size table shown in FIG.

  An image 501 illustrated in FIG. 7 indicates a compression target image. The image 501 is divided into blocks 1 to N. A compression process is performed on each block. A table 502 shows a size table. The size table 501 holds the compressed data size for each of the blocks 1 to n.

  Here, assuming that the compression unit is a macroblock unit, an image of 1920 × 1088 format is divided into 120 × 68 blocks only by luminance. Therefore, the number of entries in the size table is 8160.

  The worst compressed data size of 16 × 16 pixels by reversible compression is 257 bytes. If the value of “compressed data size-1” is stored in one entry, 8 bits = 1 byte is represented in one entry of the size table. Is done. Therefore, in one picture, the capacity required for luminance is 1 × 8160, which is about 8 Kbytes.

  On the other hand, if one pixel of a 1920 × 1088 image is 8 bits, the data size is about 2048 Kbytes only with luminance, and it can be seen that the size table has a sufficiently small capacity. The memory capacity for block 0 is a total of 193 bytes including the size of compressed data 192 bytes and 1 byte to represent 192 bytes.

  Therefore, the influence of the bandwidth and size increase of the external memory due to the size table generated by compression is considered to be a negligible level.

  Next, an example of reversible / irreversible selection information will be described. FIG. 8 is a diagram for explaining reversible / irreversible selection information. A table 601 shows an example of reversible / irreversible selection information. In the reversible / irreversible selection information 601, reversible or irreversible is set for each block of the image 501. Furthermore, in the case of irreversible, it indicates how many bits the compression is performed.

  For example, in the case of 8 bits per pixel, a block whose corresponding entry is “00” indicates lossless compression. A block whose corresponding entry is “01” indicates that irreversible compression with 8-1 = 7 bit precision is performed by reducing 1 bit indicated by the entry.

  The compression unit in the lossless compression process and the lossy compression process is arbitrary, and may be a 16 × 16 macroblock unit as the processing unit. If processing time restrictions are severe, the macroblock is divided into 16 × 4 block units. Each of them may be compressed in parallel.

  In the case of irreversible compression, it is conceivable to calculate the difference value by reducing the range of the pixel value from 8 bits to 7 bits, for example. If a higher compression ratio is desired, the range of one pixel can be lowered to 6 bits and 5 bits. As a result, when irreversible compression is performed, it is possible to prevent unnecessary compression by rounding pixel values by 1 bit.

  The reversible / irreversible selection information may be information indicating simply reversible or irreversible for one image, or may be information indicating reversible or irreversible for each block.

<Estimation processing>
Next, the estimation process performed by the estimation unit 409 will be described. First, the estimation unit 409 acquires a first size table and a second size table of past images from the memory 400. The past image is, for example, an image immediately before the current image.

  The estimation unit 409 accumulates the data size of each entry in the first size table. The estimation unit 409 accumulates the data size of each entry in the second size table.

  FIG. 9 is a diagram illustrating an example of a compression result indicating whether each block of the past image has been subjected to lossless compression or lossy compression. FIG. 9A is a compression result table indicating whether each entry in the first size table is reversible or irreversible. Each entry corresponds to each block of the past image. In the example shown in FIG. 9A, the first compression unit 402 performs lossless compression, which indicates that all entries are lossless.

  FIG. 9B is a compression result table indicating whether each entry in the second size table is reversible or irreversible. In the example shown in FIG. 9B, the second compression unit 402 performs compression based on the reversible / irreversible selection information, and therefore corresponds to the reversible / irreversible selection information when the past image is compressed.

  Here, the estimation unit 409 may accumulate only the losslessly compressed blocks in both the first size table and the second size table. Thereby, only the influence by the second image processing unit 404 can be reflected on the change in the characteristics of the image or the change in the compression efficiency. This is because if the data size of irreversible compression is included in the accumulated value, the change in image characteristics or the change in compression efficiency due to irreversible compression is also included in the change in image characteristics or the change in compression efficiency.

  For example, in the example shown in FIG. 9B, the estimation unit 409 accumulates only the data size of the entry indicating reversibility, and the accumulation corresponding to the case where all the blocks of one image are reversibly compressed from this accumulated value. Calculate the value.

  The estimation unit 409 calculates a ratio from the cumulative value of the blocks compressed by the first compression unit 402 in the past image to the cumulative value of the blocks compressed by the second compression unit 405. The estimation unit 409 multiplies the first size table of the current image by this ratio to estimate the data size after compression by the second compression unit 405.

In summary, the estimation unit 409 estimates the data size after compression of the current image by the second compression unit 405 using, for example, the following equations (1) to (3).
First reversible size = accumulated value of reversibly compressed data size in the first size table of the past image (1)
Second reversible size = cumulative value of reversibly compressed data size in the second size table of past images × (number of blocks of one image / number of blocks reversibly compressed) (2)
Estimated value of second size table of current image = each data size of first size table of current image × (second reversible size / first reversible size) Expression (3)
Accordingly, the estimation unit 409 can calculate the estimated value of the second size table of the current image when the second compression unit 405 performs lossless compression, and can control the compression by the second compression unit 405 based on the estimated value. it can.

  For example, the estimation unit 409 may perform the lossless compression by the second compression unit 405 when the accumulated value of the estimated values of the second size table does not exceed the target data size. Control to do. The estimation unit 409 controls the second compression unit 405 to perform irreversible compression when the accumulated value of the estimated values in the second size table exceeds the target data size.

  Further, the estimation unit 409 may determine lossless or lossy for each block of an image in order to prevent data loss as much as possible even when controlling to perform lossy compression. Thereby, only the minimum necessary block can be irreversibly compressed.

  Further, the estimation unit 409 may correct the estimated value using the past image data actually compressed by the second compression unit 405 in order to increase the accuracy of the estimation.

  FIG. 10 is a diagram for explaining the correction of the estimated value. FIG. 10A shows an estimated value of the second size table of the current image. FIG. 10B shows average error information obtained by calculating an average estimation error for each section of estimated values of past images.

  In the example shown in FIG. 10B, the estimation unit 409 divides into sections each having 32 bits, and calculates the average error between the estimated value corresponding to this section and the actually compressed data size. The average error information illustrated in FIG. 10B illustrates an example in which the average error increases as the estimated value increases, but is not limited thereto.

  FIG. 10C shows the estimated value of the second size table corrected using the average error information. In the example illustrated in FIG. 10C, the estimation unit 409 identifies which section of the average error information the estimated value of each entry corresponds to, and corrects the estimated value based on the average error of the section.

  The estimation unit 409 may control the compression by the second compression unit 409 using the corrected estimated value of the second size table of the current image. The estimation unit 409 generates reversible / irreversible selection information indicating whether it is reversible or irreversible according to the magnitude of the estimated value after correction. For example, the estimation unit 409 may irreversibly reduce the bit accuracy from the block with the largest estimated value, and stop the irreversible selection when it falls below the target data size. In the reversible / irreversible selection information, reversible is set for all blocks by default.

  Thereby, by having the said structure, the size of the memory which memorize | stores an image can be reduced appropriately, processing an image efficiently in a pipeline process.

<Operation>
Next, the operation of the image processing apparatus 40 in the embodiment will be described. First, the past image processing will be described. FIG. 11 is a flowchart illustrating an example of past image processing. The past image is the current image at the stage of processing.

  In step S101 shown in FIG. 11, the first image processing unit 401 performs first image processing on the input image in units of blocks. The first image processing unit 401 performs, for example, MPEG2 decoding processing.

  In step S <b> 102, the first compression unit 402 reversibly compresses one block acquired from the first image processing unit 401.

  In step S103, the control unit 408 determines whether the first image processing and the first compression processing have been performed on all blocks. If the process has been completed for all blocks (step S103—YES), the process proceeds to step S104. If the process for all blocks has not been completed (step S103—NO), the process returns to step S101.

  In step S <b> 104, the first compression unit 402 writes a first size table indicating the compressed data size of each block in the memory 400.

  In step S105, the control unit 408 notifies the first development unit 403, the second image processing unit 404, and the second compression unit 405 of a process start instruction.

  In step S <b> 106, the second compression unit 405 reads the reversible / irreversible selection information generated by the control unit 408 from the memory 400.

  In step S107, the first expansion unit 403 reads the first encoded data from the memory 400 in units of blocks and performs expansion processing. The decompression process is a decoding process corresponding to the compression encoding by the first compression unit 402.

  In step S <b> 108, the second image processing unit 404 performs second image processing on the block acquired from the first development unit 403. The second image processing unit 404 performs, for example, deblocking filter processing.

  In step S109, the second compression unit 405 performs lossless compression or lossy compression on the block acquired from the second image processing unit 404 based on the lossless / lossy selection information. At the time of irreversible compression, for example, what bit precision should be used for irreversible compression or a quantization coefficient may be set in the reversible / irreversible selection information.

  In step S110, the control unit 408 determines whether the second compression process has been completed for all blocks. If the process has been completed for all blocks (step S110—YES), the process proceeds to step S111. If the process has not been completed for all blocks (step S110—NO), the process returns to step S111.

  In step S111, the second compression unit 405 outputs the second size table indicating the compressed data size of each block and the reversible / irreversible selection information to the memory 400. The first size table, the second size table, and the reversible / irreversible selection information of the past image stored in the memory 400 are used to estimate the data size of the next image after compression by the second compression unit 405. Is done.

  FIG. 12 is a flowchart illustrating an example of a current image estimation process. In step S201 illustrated in FIG. 12, the control unit 408 determines whether the compression processing by the second compression unit 405 of the past image has been completed. If the process has been completed (step S201—YES), the process proceeds to steps S202 and 203. If the process has not been completed (step S201—NO), the process returns to step S201.

  In step S <b> 202, the estimation unit 409 reads the first size table of the past image from the memory 400.

  In step S203, the estimation unit 409 calculates the total size (also referred to as lossless synthesis size) of the read first size table entries.

  In step S <b> 204, the estimation unit 409 reads the second size table of the past image and the reversible / irreversible selection information from the memory 400.

  In step S205, the estimation unit 409 adds up the data sizes of the second size table of entries indicating reversibility among the reversible / irreversible selection information. In addition, the estimation unit 409 calculates the number of reversible blocks.

  In step S206, the estimation unit 409 calculates the total estimated value (also referred to as an estimated reversible total size) in the second size table when all past images are reversibly compressed. The estimated reversible size is calculated by the above-described equation (2).

  In addition, although the process of step S202-S203 and the process of step S204-S206 were described in parallel, it does not necessarily need to be a parallel process.

  In step S207, the estimation unit 409 calculates a change ratio between the reversible total size calculated in step S203 and the estimated reversible total size calculated in step S206. Thereby, a change in image characteristics or a change in compression efficiency by the second image processing unit 404 can be obtained.

  In step S208, the control unit 408 determines whether or not the current image has been reversibly compressed by the first compression unit 402. If the process has been completed (step S208—YES), the process proceeds to step S209. If the process has not been completed (step S208—NO), the process returns to step S208.

  In step S209, the estimation unit 409 reads the first size table of the current image from the memory 400.

  In step S210, the estimation unit 409 corrects the first size table of the current image by multiplying by the change ratio calculated in step S207. The first size table multiplied by the change ratio is called an estimation table.

  In step S211, the estimation unit 409 calculates the total size of the entries in the current image estimation table.

  In step S212, the estimation unit 409 determines whether the total size is equal to or smaller than the target size. If the total size is less than or equal to the target (step S212—YES), the process proceeds to step S213, and if the total size is larger than the target (step S212—NO), the process proceeds to step S214.

  In step S213, the estimation unit 409 sets all the reversible / irreversible selection information entries of the current image to “00” and generates reversible / irreversible selection information. Here, “00” indicates reversibility. That the total value is equal to or smaller than the target size indicates that all blocks may be reversibly compressed.

  In step S214, the estimation unit 409 performs reversible / irreversible selection information generation processing. This generation process will be described later.

  In step S215, the estimation unit 409 writes the generated lossless / irreversible selection information in the memory 400.

  In step S216, the control unit 408 notifies the second image processing unit 404 of a start instruction for the second image processing. Thereby, the second image processing and the second compression processing are performed on the current image.

  FIG. 13 is a flowchart illustrating an example of processing for generating reversible / irreversible selection information. In step S301 illustrated in FIG. 13, the estimation unit 409 calculates an error between the second size table of the past image and the estimation table.

  In step S302, the estimation unit 409 generates average error information using the calculated error. For example, the estimation unit 409 calculates an average error of the actually compressed data size with respect to the estimated data size for each unit section. For example, if compression is performed in units of 16 × 16 pixels, a unit section of 16 bytes is set.

  In step S303, the estimation unit 409 corrects each entry in the estimation table for the current image based on the average error information.

  In step S304, the estimation unit 409 determines whether the total value of each entry in the corrected estimation table is equal to or less than the target size. If the total value is less than or equal to the target size (step S304—YES), the process proceeds to step S305, and if the total value is greater than the target size (step S304—NO), the process proceeds to step S306.

  In step S305, the estimation unit 409 sets all the reversible / irreversible selection information entries of the current image to “00” and generates reversible / irreversible selection information. That the total value is equal to or smaller than the target size indicates that all blocks may be reversibly compressed.

  In step S306, the estimation unit 409 performs a process of selecting reversible or irreversible for each block of the original image, and generates reversible / irreversible selection information. Thereby, the compression of the second compression unit 405 can be controlled by the estimated value with the estimated accuracy increased.

  FIG. 14 is a flowchart illustrating an example of reversible / irreversible selection processing. In the processing shown in FIG. 14, the area having the largest size is selected from the entries in the estimation table, and the area is irreversibly compressed.

  The reason why the large-size region is made irreversible is that the estimated size is large, because there are many high-frequency components, so that the compression efficiency can be expected to be poor. In general, since the human eye is insensitive to high-frequency components, it is considered that the influence on the overall image quality is small even if the range of one pixel in the target area is lowered.

In step S401 illustrated in FIG. 14, the estimation unit 409 selects the entry with the largest size.
cur_sft = reversible / irreversible selection information entry corresponding to the largest size entry cur_size = largest size estimation table entry In step S402, the estimation unit 409 drops the block range of the largest size estimation value by 1 bit. The value of the entry with the largest size is updated by the following formula.
Entry of reversible / irreversible selection information corresponding to the largest entry = cur_sft + 1 (drops the range by 1 bit)
Entry of estimation table having the largest size = cur_size × ((8−cur_sft−1) / (8−cur_sft))
In step S403, the estimation unit 409 determines whether the total value of all entries in the updated estimation table is equal to or less than the target size. If the total value is less than or equal to the target size (step S403-YES), the reversible / irreversible selection process ends, and if the total value is larger than the target size (step S403-NO), the process returns to step S401.

  As a result, the range of the largest entry can be reduced by 1 bit. For example, when the range is decreased from 8 bits to 7 bits per pixel, it can be expected that the compressed data size is the data amount corresponding to the dropped bits, that is, the compressed data size is 7/8.

  Therefore, the estimated table entry is updated by multiplying the estimated size value by 7/8. By repeating this process until the total value in the estimation table falls within the target size, final reversible / irreversible selection information can be obtained. It is assumed that “00” is set as the initial value in all entries of the reversible / irreversible selection information.

  FIG. 15 is a flowchart illustrating another example of the reversible / irreversible selection process. The process shown in FIG. 15 is a process for selecting the lossy compression process if it does not fall within the target code amount for each block.

  In step S501 illustrated in FIG. 15, the estimation unit 409 substitutes 0 for cur_sft and sets the initial value of each entry of the reversible / irreversible selection information to 0.

  In step S502, the estimation unit 409 selects blocks (entries) in the raster scan order, for example, from the estimation table. The selected block is called the current block. The estimation unit 409 substitutes the estimated size of the current block for cur_size.

  In step S503, the estimation unit 409 determines whether cur_size is equal to or smaller than the target size. At this time, the target size is a value obtained by dividing the target size of one image by the number of blocks in one image.

  If cur_size is equal to or smaller than the target size (step S503-YES), the process proceeds to step S506, and if cur_size is larger than the target size (step S503-NO), the process proceeds to step S504.

  In step S504, the estimation unit 409 increments cur_sft by 1 to reduce the range by 1 bit.

In step S505, the estimation unit 409 updates the entry value for the current block using the following equation.
Entry of reversible / irreversible selection information corresponding to entry of current block = cur_sft
Estimate table entry corresponding to current block entry = cur_size × ((8−cur_sft) / (8−cur_sft + 1))
After step S505, the process returns to step S503. As a result, the range becomes smaller by 1 bit until the entry value (estimated size) of the current block becomes equal to or smaller than the target size.

  In step S506, the estimation unit 409 determines whether scanning of all blocks has been completed. If the scanning of all blocks is completed (step S506-YES), the reversible / irreversible selection process is completed, and if the scanning of all blocks is not completed (step S506-NO), the process returns to step S502. Another block is selected.

  As a result, the process shown in FIG. 15 does not require scanning to detect the largest block among the blocks, so that the reversible / irreversible selection information can be generated only by the sequential processing loop. Reduction is possible.

  As described above, according to the embodiment, it is possible to appropriately reduce the size of a memory for storing an image while efficiently processing the image in pipeline processing. This is because the data size after compression by the second compression unit can be estimated appropriately. Further, according to the embodiment, precise rate control is possible. In addition, according to the embodiment, due to inaccuracy of estimation, the lossy compressed data size greatly exceeds the target size, or conversely, the lossy area is excessively increased, resulting in image quality deterioration. In other words, it is possible to implement only the necessary minimum lossy processing.

  Note that since the image characteristics change when the output image of the first image processing unit is processed by the second image processing unit, two-pass processing using the statistical information obtained by the first compression unit is simply used. Then the effect is weak.

  Therefore, in the embodiment, using the fact that moving images have similar image characteristics of images close in time, statistical information obtained when the past image is compressed by the first and second compression units is used. Use. In the embodiment, the size table generated when the current image is compressed by the first compression unit is corrected, and an estimated size table for the second compression unit is generated. Therefore, in the embodiment, it can be expected to obtain an estimation result with higher accuracy than simple two-pass processing.

  In the above-described embodiment, the MPEG2 decoding process and the deblocking filter process are taken as an example. However, the present invention is not limited to the contents of the embodiment, and can be applied to a system LSI for arbitrary moving image processing in which a plurality of image processing circuits sequentially process in picture level pipeline processing.

  In the embodiment, the two-stage processing is performed. However, even when a plurality of stages of moving image processing of three or more stages are linked, it can be applied to the compression unit after each process. In the embodiment, the estimation process is executed by the control unit 408. However, depending on the design constraints, the estimation process may be executed as dedicated hardware, or the estimation process may be executed inside the second compression unit 405. .

  Although the embodiment has been described in detail above, the present invention is not limited to this embodiment, and various modifications and changes other than the above-described embodiment and modifications can be made within the scope described in the claims. is there.

In addition, the following additional remarks are disclosed regarding the above Example.
(Appendix 1)
An image processing apparatus that sequentially processes the processes of a first image processing unit and a second image processing unit,
A first compression unit that compresses an image output from the first image processing unit;
A second compression unit that compresses an image output from the second image processing unit;
Using the data size of the current image compressed by the first compression unit and the ratio of the data size of the past image respectively compressed by the first and second compression units, the data after compression by the second compression unit A control unit that estimates a data size of the current image and controls compression of the current image by the second compression unit based on the estimated data size;
An image processing apparatus comprising:
(Appendix 2)
The first compression unit includes
Reversibly compresses the image into blocks,
The second compression unit includes
The image processing apparatus according to appendix 1, wherein lossless compression or lossy compression is performed for each block.
(Appendix 3)
The controller is
Based on the estimated data size, for each block of the current image, generating selection information indicating reversible or irreversible,
The second compression unit includes
The image processing apparatus according to appendix 2, wherein reversible or irreversible is selected and compressed for each block of the current image based on the selection information.
(Appendix 4)
The controller is
When estimating the data size of the current image after compression by the second compression unit, the second compression unit using a reversibly compressed block among the blocks of the past image compressed by the second compression unit 4. The image processing apparatus according to appendix 2 or 3, which calculates a data size of the past image after compression by.
(Appendix 5)
The controller is
Additional note for correcting the estimated data size of the current image using an error between the estimated data size of the past image and the data size of the past image compressed by the second compression unit The image processing apparatus according to any one of 1 to 4.
(Appendix 6)
The controller is
The image processing according to appendix 3, wherein when the estimated value obtained by multiplying the data size of the current image compressed by the first compression unit by the ratio is larger than a threshold value, irreversible is selected for a predetermined block of the current image. apparatus.
(Appendix 7)
The controller is
The image processing apparatus according to appendix 6, wherein the data size after compression by the second compression unit is estimated for each block of the current image, and the predetermined block is determined based on the data size estimated for each block.
(Appendix 8)
An image processing method for sequentially processing the first image processing unit and the second image processing unit,
Compressing the current image output from the first image processing unit;
When compressed after the second image processing unit by using the compressed data size of the current image and the ratio of the data size of the past image compressed after the first and second image processing units, respectively. Generating the control information for the compression of the current image output from the second image processing unit based on the estimated data size,
An image processing method for compressing a current image output from the second image processing unit based on the control information.

40 image processing device 400 memory 401 first image processing unit 402 first compression unit 403 first expansion unit 404 second image processing unit 405 second compression unit 406 second expansion unit 407 output unit 408 control unit 409 estimation unit

Claims (6)

An image processing apparatus that pipelines the processing of a first image processing unit and a second image processing unit,
A first compression unit that compresses a current image output from the first image processing unit and stores the compressed image in a storage unit ;
A decompression unit that decompresses the data compressed by the first compression unit read from the storage unit and supplies the decompressed data to the second image processing unit;
A second compression unit that compresses an image output from the second image processing unit;
Using the data size of the current image compressed by the first compression unit and the ratio of the data size of the past image respectively compressed by the first and second compression units, the data after compression by the second compression unit A control unit that estimates a data size of the current image and controls compression of the current image by the second compression unit based on the estimated data size;
An image processing apparatus comprising: The first compression unit includes
Reversibly compress the current image for each divided block,
The second compression unit includes
The image processing apparatus according to claim 1, wherein lossless compression or lossy compression is performed for each block. The controller is
Based on the estimated data size, for each block of the current image, generating selection information indicating reversible or irreversible,
The second compression unit includes
The image processing apparatus according to claim 2, wherein reversible or irreversible is selected and compressed for each block of the current image based on the selection information. The controller is
When estimating the data size of the current image after compression by the second compression unit, the second compression unit using a reversibly compressed block among the blocks of the past image compressed by the second compression unit The image processing apparatus according to claim 2, wherein the data size of the past image after compression by the method is calculated. The controller is
The correction of the estimated data size of the current image using an error between the estimated data size of the past image and the data size of the past image compressed by the second compression unit. Item 5. The image processing apparatus according to any one of Items 1 to 4. An image processing method for pipeline processing the processing of a first image processing unit and a second image processing unit,
Compressing the current image output from the first image processing unit and storing it in a storage unit ;
Expanding the compressed data of the current image read from the storage unit and supplying it to the second image processing unit,
Compressing the image output from the second image processing unit;
When compressed after the second image processing unit by using the compressed data size of the current image and the ratio of the data size of the past image compressed after the first and second image processing units, respectively. Generating the control information for the compression of the current image output from the second image processing unit based on the estimated data size,
An image processing method for compressing a current image output from the second image processing unit based on the control information.

Images