JP2009147751A - Image processor and imaging apparatus loaded with the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem of fluctuation in image quality such as flickering in the trend of high image quality and high compression ratio. <P>SOLUTION: An image processor 200 encodes a moving image. The image processor 200 includes a preprocessing part 50 and an in-frame predicted encoding part 60. The preprocessing part 50 applies a preprocess for partially succeeding the image quality of encoded image, having been subjected to inter-frame encoding, which proceeds in terms of time to the image to be subjected to in-frame predicted encoding. The in-frame predicted encoding part 60 subjects a target image having been preprocessed by the preprocessing part 50 to the in-frame predicted encoding. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that encodes a moving image using an intra-frame predictive coding technique and an inter-frame predictive coding technique, and an imaging apparatus equipped with the image processing apparatus.

  Digital movie cameras are becoming popular. Digital movie cameras have improved image quality year by year, and those compatible with full HD (high definition) image quality have also been put into practical use. As a result, the image compression efficiency has also increased. Those capable of compression coding according to the H.264 / AVC standard have been put into practical use.

Patent Document 1 discloses an image quality improving apparatus. This image quality improvement apparatus detects a level fluctuation of a decoded image of a signal encoded by an encoding method using bidirectional prediction, and improves the decoded image quality based on the level fluctuation. Patent Document 2 discloses an encoding device. This encoding apparatus calculates an activity indicating the flatness of an image based on image data for each macroblock, and corrects a cost value based on a cost function indicating encoding efficiency based on the activity. Based on this, the optimum mode is detected for each macroblock from a plurality of intra prediction modes and a plurality of inter prediction modes.
JP-A-9-224250 JP 2006-94081 A

  As described above, with the progress of higher image quality and higher compression, the problem of image quality fluctuations such as flicker has become more obvious and has affected subjective image quality. In particular, flicker that occurs when switching from an image subjected to inter-frame predictive coding to an image subjected to intra-frame predictive coding is conspicuous.

  A major cause of the occurrence of flicker is a quantization error. The quantization error is an error generated by quantizing an orthogonal transform coefficient such as a DCT (Discrete Cosine Transform) coefficient. Increasing the quantization scale at the time of quantization can improve the compression efficiency, but increases the quantization error. In the MPEG series encoded stream, as long as the P or B picture continues, the influence of the quantization error of the reference picture is dragged, so that the quantization error is accumulated. Therefore, when an I picture appears, the I picture is not affected by the reference picture, and the accumulated quantization error is reset. The resetting of the quantization error that occurs when switching from the P or B picture to the I picture is a major cause of flicker.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus capable of suppressing fluctuations in image quality and an imaging apparatus equipped with the image processing apparatus.

  An aspect of the present invention is an image processing apparatus that encodes a moving image, in which a target image to be subjected to intraframe predictive encoding is obtained by encoding an interframe encoded encoded image that temporally precedes the image. A pre-processing unit that performs pre-processing for taking over part of the image quality; and an intra-frame prediction encoding unit that performs intra-frame prediction encoding on the target image that has been pre-processed by the pre-processing unit.

  It should be noted that any combination of the above-described constituent elements and a representation obtained by converting the expression of the present invention between a method, an apparatus, a system, a program and the like are also effective as an aspect of the present invention.

  According to the present invention, image quality fluctuation can be suppressed.

  FIG. 1 is a diagram illustrating a configuration of an imaging apparatus 300 equipped with an image processing apparatus 200 according to an embodiment. The imaging device 300 includes an imaging unit 100 and an image processing device 200. The image processing apparatus 200 includes a frame image dividing unit 20, an orthogonal transform unit 30, a quantization unit 40, a preprocessing unit 50, an intra-frame prediction encoding unit 60, an inter-frame prediction encoding unit 70, a variable length encoding unit 80, And a buffer 90.

  The configuration of the image processing apparatus 200 can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and is realized in software by a program having an image encoding function loaded in the memory. Here, however, the functional blocks realized by the cooperation are depicted. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The imaging unit 100 acquires a moving image and supplies it to the image processing apparatus 200. The imaging unit 100 includes an imaging element (not shown) such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and a signal processing circuit (not shown) that processes a signal output from the element. The signal processing circuit can convert the analog three primary color signals R, G, and B output from the image sensor into digital luminance signals Y and color difference signals Cr and Cb.

  The image processing apparatus 200 processes the moving image acquired by the imaging unit 100. More specifically, the moving image is compression-encoded according to a predetermined standard. For example, H.M. H.264 / AVC, MPEG-2, MPEG-4, etc. are compressed and encoded.

  The frame image dividing unit 20 divides each frame image into a plurality of regions. Hereinafter, in this embodiment, it is assumed to be divided into macro blocks. The orthogonal transform unit 30 performs orthogonal transform on each frame image in units of macro blocks. In the MPEG series, the luminance signal Y and the color difference signals Cr and Cb are DCT transformed to generate DCT coefficients.

The quantization unit 40 quantizes the DCT coefficient generated by the orthogonal transform unit 23 with reference to a predetermined quantization table. The quantization table defines a quantization scale to which each DCT coefficient should be divided. Of the DCT coefficients, the quantization scale corresponding to the low frequency component is set small, and the quantization scale corresponding to the high frequency component is set large. Thereby, omission can be enlarged as a high frequency component. Note that all or some of the quantization scales defined in the quantization table are adaptively variably controlled.
The quantization unit 40 supplies an image to be subjected to intraframe prediction encoding to the preprocessing unit 50. On the other hand, an image to be subjected to interframe prediction encoding is supplied to the interframe prediction encoding unit 70.

The pre-processing unit 50 performs pre-processing on the target image to be subjected to intra-frame prediction encoding so as to take over part of the image quality of the encoded image that is temporally preceded by the inter-frame encoding. The inherited image quality includes an influence on the image quality generated by interframe coding. For example, it takes over the influence of high frequency components such as quantization errors being cut off.
Details of the preprocessing will be described later.
The intra-frame prediction encoding unit 60 performs intra-frame prediction encoding on the target image that has been pre-processed by the pre-processing unit 50. Here, intra-frame predictive coding defined in the MPEG series can be used.

  The inter-frame predictive encoding unit 70 performs inter-frame predictive encoding on the quantized image. Again, inter-frame predictive coding defined in the MPEG series can be used. That is, a motion compensation technique can be used. Specifically, the inter-frame prediction encoding unit 70 searches for a prediction area with the smallest error from the past or future reference images for each macroblock, and obtains a motion vector indicating a shift to the prediction area. Then, motion compensation is performed for each macroblock using the motion vector, and a prediction error signal is generated. Note that the motion compensation processing is performed not on the DCT coefficient but on the pixel value, but here, in order to simplify the description, in FIG. 1, the system for inverse quantization and inverse orthogonal transformation is omitted.

  The variable length coding unit 80 entropy codes the DCT coefficient, the prediction error signal, the motion vector, and other parameters generated by the intraframe prediction coding unit 60 and the interframe prediction coding unit 70. In this way, the moving image acquired by the imaging unit 100 is compression encoded and converted into an encoded stream.

  The buffer 90 temporarily stores the encoded stream, and records the encoded stream on a recording medium (not shown) such as a memory card, a hard disk, or a DVD at a predetermined timing, or sends the encoded stream to a network. Further, the buffer 90 supplies the quantization unit 40 with the code amount of the encoded stream or the buffer occupation amount by the encoded stream. These pieces of information are used as judgment materials when determining the quantization scale. Further, these pieces of information can also be supplied to the switching unit 55 described later.

Hereinafter, various embodiments of the preprocessing unit 50 will be described.
FIG. 2 is a diagram illustrating the configuration of the preprocessing unit 50 according to the first embodiment. The preprocessing unit 50 according to the first embodiment includes an inter-frame prediction encoding unit 52 and a decoding unit 53.
The inter-frame predictive encoding unit 52 performs inter-frame predictive encoding on the target image to be intra-frame predictive encoded using a decoded image of the inter-frame encoded encoded image that precedes the image in time as a reference image. . The target image is supplied from the quantization unit 40, and the encoded image that has been inter-frame encoded is supplied from the inter-frame prediction encoding unit 70.

  In the MPEG series, the target picture to be subjected to intraframe prediction encoding is an I picture. The inter-frame encoded image that precedes the image in time is a P picture or B picture that precedes the I picture in time. The reference image is preferably an image immediately before the target image, but an image before that may be used. In particular, when the image immediately before the target image is a B picture, the P picture closest to the target image in the past direction may be used as the reference image. Since the P picture more accurately inherits the quantization error along the time flow than the B picture, the image quality variation along the time flow can be taken into the target image more faithfully.

  The inter-frame prediction encoding unit 52 is drawn in the preprocessing unit 50 for easy understanding of the explanation, but naturally, the inter-frame prediction encoding unit 70 can share hardware resources and software resources. In addition, an encoded image that has been inter-frame encoded by the inter-frame predictive encoding unit 70 becomes a decoded image through inverse quantization, inverse orthogonal transform, and motion compensation. The image is drawn so as to be supplied to the inter-frame prediction encoding unit 52 as it is.

  The decoding unit 53 decodes the encoded image encoded by the interframe prediction encoding unit 52 and passes the decoded image to the intraframe prediction encoding unit 60. Specifically, the encoded image is subjected to inverse quantization, inverse orthogonal transform, and motion compensation to obtain a decoded image, which is a corrected image that partially inherits the image quality of the temporally preceding image. This is supplied to the inner prediction encoding unit 60.

  As described above, according to the first embodiment, an image to be subjected to interframe prediction encoding is performed by preliminarily performing interframe prediction encoding on an image to be subjected to intraframe prediction encoding and performing intraframe prediction encoding on the decoded image. And fluctuations in image quality such as flicker that occur while switching between intra-frame predictive-encoded images can be suppressed. In other words, by preliminarily inter-frame predictively encoding an image to be intra-frame predictive encoded, it is possible to partially inherit the quantization error etc. of the preceding image, and image quality fluctuations caused by resetting the quantization error etc. Can be reduced. Therefore, the subjective image quality can be improved.

  Further, the first embodiment is a simple process compared to a method of taking over the image quality of the preceding image in units of macroblocks to be described later. This is effective when the processing load is large or for low-spec devices.

FIG. 3 is a diagram illustrating the configuration of the preprocessing unit 50 according to the second embodiment. The preprocessing unit 50 according to the second embodiment includes a reference image correction unit 51, an inter-frame prediction encoding unit 52, and a decoding unit 53.
The reference image correction unit 51 corrects a part of the encoded image temporally preceding the target image and has a strong correlation with the encoded image further preceding the encoded image. To the inter-frame prediction encoding unit 52 as the reference image. Here, the partial area having a strong correlation may be a macroblock whose motion vector is smaller than the first threshold between decoded images of both encoded images. Further, the macro block may have a prediction error smaller than the second threshold. Alternatively, it may be a macroblock that satisfies both conditions. Note that the correlation may be determined in units of areas larger or smaller than the macroblock. The first threshold value and the second threshold value can be set based on experiments and simulations by the designer. Details of the reference image correction unit 51 will be described later. The inter-frame prediction encoding unit 52 and the decoding unit 53 are basically the same as those in the first embodiment. The difference is that the reference image supplied to the inter-frame prediction encoding unit 52 becomes a reference image corrected by the reference image correcting unit 51.

  As described above, according to the second embodiment, the same effects as the first embodiment are obtained. Although details will be described later, it is possible to further reduce high-frequency noise and improve compression efficiency as compared with the first embodiment.

  FIG. 4 is a diagram illustrating the configuration of the preprocessing unit 50 according to the third embodiment. The preprocessing unit 50 according to the third embodiment includes a target image correction unit 54. The target image correction unit 54 corrects a part of the target image to be intra-frame encoded that is temporally preceded by the image and has a strong correlation with the inter-frame encoded image. The image is passed to the intraframe prediction encoding unit 60. More specific description will be given below.

The target image correction unit 54 includes a motion compensation unit 541, a block extraction unit 542, an intermediate image generation unit 545, and an image assembly unit 546.
The motion compensation unit 541 searches for a motion in the decoded image of the encoded image for each of a plurality of divided blocks in the target image to obtain a motion vector and a predicted image. Specifically, the motion compensation unit 541 identifies, for each macroblock in the target image, the region with the smallest difference within a predetermined motion search range in the reference image, using the decoded image as a reference image. To do. For example, a region where the sum of absolute values and sum of squares of prediction errors between corresponding pixels is the smallest is specified. The motion compensation unit 541 obtains a motion vector between each macroblock and each identified area, and supplies the motion vector to the block extraction unit 542. In addition, the image of each specified area is supplied to the block extraction unit 542 as a predicted image.

  The block extraction unit 542 extracts a block whose motion vector is smaller than a first threshold and whose difference from the predicted image is smaller than a second threshold among a plurality of divided blocks in the target image. Thereby, it is possible to identify a region that matches or substantially matches between the target image and the reference image that precedes in time, and that region can be a region that should inherit the image quality of the reference image.

  The intermediate image generation unit 545 generates an intermediate image between the original image of the block extracted by the block extraction unit 542 and the predicted image corresponding thereto. For example, an average value of the pixel values of the original image and the predicted image may be calculated to generate the intermediate image. In addition, the above calculation may be executed after weighting at least one pixel value of the original image and the predicted image with a predetermined coefficient.

  The image assembling unit 546 generates the corrected image by replacing the intermediate image generated by the intermediate image generating unit 545 with the original image corresponding to the intermediate image in the target image. This corrected image is a target to be subjected to intraframe predictive encoding in the intraframe predictive encoding unit 60.

  FIG. 5 is a diagram illustrating how corrected images are generated according to the third embodiment. In FIG. 5, the target image Fs to be subjected to intraframe prediction encoding, the preceding images Fp1 and Fp2, and the reference image Fr temporally preceding the target image Fs and subjected to interframe prediction encoding are depicted. Yes. In FIG. 5, the preceding image immediately before the target image Fs is set as the reference image Fr. A corrected image Fc is generated based on the target image Fs and the reference image Fr. Regions I1 and I2 in the corrected image Fc are regions that are replaced with intermediate images that inherit the image quality of the reference image Fr.

FIG. 6 is a flowchart illustrating a procedure of image processing according to the third embodiment.
First, the motion compensation unit 541 performs motion compensation for each macroblock of the target image, and specifies each predicted image in the target image (S10). Next, the block extraction unit 542 sets 1 as an initial value in the parameter data n (S11).

  The block extraction unit 542 determines whether the motion vector of the macroblock MB (n) is lower than the first threshold value and the difference between the original image of the macroblock MB (n) and the corresponding predicted image is lower than the second threshold value. It is determined whether or not (S12). When both conditions are satisfied (Y in S12), the block extraction unit 542 extracts the original image of the macroblock MB (n) and the corresponding prediction image (S13), and the intermediate image generation unit 545 An intermediate image having intermediate image quality is generated (S14). Then, the block extraction unit 542 increments the parameter n (S15). If at least one condition is not satisfied in step S12 (N in S12), the parameter n is incremented without executing the processes in steps S13 and S14 (S15).

  The block extraction unit 542 determines whether or not the parameter n exceeds the number of macroblocks (S16). If the parameter n does not exceed (N in S16), the process proceeds to step S12 and the above-described processing is repeated. If it exceeds (Y in S16), the image assembling unit 546 generates a corrected image using the intermediate image (S17). Specifically, an image in a region corresponding to the intermediate image in the target image is replaced with the intermediate image. The intraframe prediction encoding unit 60 performs intraframe prediction encoding on the corrected image (S18).

  As described above, according to the third embodiment, an image to be interframe prediction encoded and an intraframe prediction code are obtained by replacing a part of an image to be intraframe prediction encoded with an image that inherits the image quality of the preceding image. It is possible to suppress fluctuations in image quality such as flicker that occur while the image to be converted is switched. In other words, a part of the quantization error of the preceding image can be taken over, and image quality fluctuations caused by resetting the quantization error can be reduced. Therefore, the subjective image quality can be improved. In addition, compared with the first and second embodiments described above and the first and second modifications of the third embodiment described later, the process is simplified because it is not necessary to perform interframe predictive encoding and decoding the target image. be able to. In the basic example of the third embodiment, the motion compensation process can be stopped among all the steps of inter-frame predictive coding.

  FIG. 7 is a diagram illustrating a configuration of the preprocessing unit 50 according to the first modification of the third embodiment. The preprocessing unit 50 according to this modification includes a target image correction unit 54. The target image correction unit 54 corrects a part of the target image to be intra-frame encoded that is temporally preceded by the image and has a strong correlation with the inter-frame encoded image. The image is passed to the intraframe prediction encoding unit 60. More specific description will be given below.

The target image correction unit 54 includes a motion compensation unit 541, a block extraction unit 542, an inter-frame prediction encoding unit 543, a decoding unit 544, an intermediate image generation unit 545, and an image assembly unit 546.
The motion compensation unit 541 searches for a motion in the decoded image of the encoded image for each of a plurality of divided blocks in the target image to obtain a motion vector and a predicted image. The block extraction unit 542 extracts a block whose motion vector is smaller than the first threshold and whose difference from the predicted image is smaller than the second threshold among the plurality of divided blocks in the target image.

  The inter-frame predictive encoding unit 543 performs inter-frame predictive encoding on at least the block extracted by the block extracting unit 542 using the decoded image as a reference image. That is, prediction error signals may be generated only for the extracted macroblocks, or prediction error signals may be generated for all macroblocks.

  The decoding unit 544 decodes the block encoded by the inter-frame prediction encoding unit 533. Again, only the macroblocks extracted by the block extraction unit 542 may be decoded, or when prediction error signals are generated for all macroblocks, all macroblocks may be decoded.

  The intermediate image generation unit 545 generates an intermediate image between the block image extracted by the block extraction unit 542 and decoded by the decoding unit 544 and the corresponding original image. Here, the original image refers to a target image that is not inter-frame encoded. The image assembling unit 546 generates the corrected image by replacing the intermediate image generated by the intermediate image generating unit 545 with the original image corresponding to the intermediate image in the target image.

FIG. 8 is a flowchart illustrating a procedure of image processing according to the first modification of the third embodiment.
The flowchart of FIG. 8 is basically the same as the flowchart of FIG. Only the object extracted in step S13 is different. In the present modification, the block extraction unit 542 extracts the original image of the macroblock MB (n) and the corresponding decoded image (S13 ′), and the intermediate image generation unit 545 has an intermediate image quality of those intermediate images. An image is generated (S14). Here, the decoded image is a partial image of an area corresponding to the macroblock MB (n) encoded by the inter-frame prediction encoding unit 543 and decoded by the decoding unit 544.

  As described above, according to the first modification of the third embodiment, by replacing a part of an image to be intraframe prediction encoded with an image that inherits the image quality of the preceding image, It is possible to suppress fluctuations in image quality such as flicker that occurs while switching between images to be subjected to intraframe prediction encoding. Also, compared to the basic example of the third embodiment, by using an intermediate image based on the decoded image of the target image as a replacement image, high-frequency noise can be reduced, and compression efficiency can be improved. Can do.

  FIG. 9 is a diagram illustrating a configuration of the preprocessing unit 50 according to the second modification of the third embodiment. The preprocessing unit 50 according to this modification includes a target image correction unit 54. The target image correction unit 54 corrects a part of the target image to be intra-frame encoded that is temporally preceded by the image and has a strong correlation with the inter-frame encoded image. The image is passed to the intraframe prediction encoding unit 60. More specific description will be given below.

The target image correction unit 54 includes a motion compensation unit 541, a block extraction unit 542, an interframe prediction encoding unit 543, a decoding unit 544, and an image assembly unit 546.
The motion compensation unit 541 searches for a motion in the decoded image of the encoded image for each of a plurality of divided blocks in the target image to obtain a motion vector and a predicted image. The block extraction unit 542 extracts a block whose motion vector is smaller than the first threshold and whose difference from the predicted image is smaller than the second threshold among the plurality of divided blocks in the target image.

  The inter-frame predictive encoding unit 543 performs inter-frame predictive encoding on at least the block extracted by the block extracting unit 542 using the decoded image as a reference image. The decoding unit 544 decodes the block encoded by the inter-frame prediction encoding unit 543.

  The image assembling unit 546 generates the corrected image by replacing the block image extracted by the block extracting unit 542 and decoded by the decoding unit 544 with the original image corresponding to the image of the block in the target image. To do. In this modified example, the intermediate image is not generated and replaced, but the decoded image itself extracted by the block extraction unit 542 and decoded by the decoding unit 544 is replaced with the original image.

FIG. 10 is a flowchart illustrating a procedure of image processing according to the second modification of the third embodiment.
The flowchart of FIG. 10 is basically the same as the flowchart of FIG. The difference is that step S14 is deleted and step S17 is changed. For the latter, the image assembling unit 546 generates a corrected image using the decoded image (S17 ′). Specifically, an image in a region corresponding to the decoded image in the target image is replaced with the decoded image.

  As described above, according to the second modification of the third embodiment, by replacing a part of an image to be subjected to intraframe prediction encoding with an image that inherits the image quality of the preceding image, It is possible to suppress fluctuations in image quality such as flicker that occurs while switching between images to be subjected to intraframe prediction encoding. Further, compared with the first modification of the third embodiment, by using the decoded image itself of the target image as a replacement image, it is possible to further reduce high-frequency noise and further improve the compression efficiency.

FIG. 11 is a diagram illustrating the configuration of the preprocessing unit 50 according to the fourth embodiment. The preprocessing unit 50 according to the present embodiment includes a reference image correction unit 51 and a target image correction unit 54.
The reference image correction unit 51 corrects a part of the encoded image that temporally precedes the target image with a strong correlation with the encoded image that temporally precedes the encoded image.
The reference image correction unit 51 can use the same technique as that of the target image correction unit 54 of the basic example of the third embodiment, the first modification, and the second modification described above. In that case, the “target image” in the third embodiment was encoded with “the encoded image temporally preceding the target image” in the fourth embodiment and the “interframe encoded temporally preceding the target image” in the third embodiment. The “encoded image” is read as “the encoded image temporally preceding the encoded image temporally preceding the target image” in the fourth embodiment.

  The target image correction unit 54 corrects a part of the target image having a strong correlation with the corrected image corrected by the reference image correction unit 51, and passes the corrected image to the intra-frame prediction encoding unit 60. . A configuration and operation similar to any of the basic example, the first modification, and the second modification of the third embodiment described above can be employed.

  FIG. 12 is a diagram illustrating how corrected images are generated according to the fourth embodiment. In FIG. 12, the target image Fs to be subjected to intraframe prediction encoding and the preceding images Fp1, Fp2, and Fp3 temporally preceding the target image Fs and subjected to interframe prediction encoding are depicted. In FIG. 12, an intermediate image between the preceding image Fp3 immediately before the target image Fs and the immediately preceding image Fp2 is used as the reference image Fr. A corrected image Fc is generated based on the target image Fs and the reference image Fr. The regions I1 and I2 in the corrected image Fc are regions that are replaced with intermediate images that inherit the image quality of the preceding images Fp2 and Fp3.

  As described above, according to the fourth embodiment, by replacing a part of an image to be subjected to intraframe predictive encoding with an image that inherits the image quality of a plurality of preceding images, an image to be interframe predictively encoded and an intraframe It is possible to suppress fluctuations in image quality such as flicker that occurs while switching between predictive-encoded images. Further, compared with Example 3, the correlation in the low frequency region between the target image and the reference image can be further strengthened, and unnecessary high frequency noise can be further reduced. At the same time, the compression efficiency can be further improved.

  FIG. 13 is a diagram illustrating the configuration of the preprocessing unit 50 according to the fifth embodiment. The preprocessing unit 50 according to the present embodiment includes a target image whole correction unit 54a, a target image partial correction unit 54b, and a switching unit 55.

  The entire target image correction unit 54a performs inter-frame predictive encoding on the target image using a decoded image of the encoded image temporally preceding the image as a reference image, decodes the encoded image, and performs intra-frame predictive encoding. Pass to part 60. The same configuration and operation as those of the inter-frame prediction encoding unit 52 and the decoding unit 53 according to the first embodiment described above can be employed.

  The target image partial correction unit 54 b corrects a part of the target image that has a strong correlation with the encoded image that precedes the image in time, and supplies the corrected image to the intra-frame prediction encoding unit 60. hand over. Any one of the configurations and operations of the basic example, the first modification, and the second modification of the third embodiment described above can be employed.

  The switching unit 55 includes whether to enable the entire target image correction unit 54a or the target image partial correction unit 54b in the processing load of the image processing apparatus 200 and the moving image to be processed by the apparatus. Switching is performed based on at least one of the correlation degrees between the frame images. You may switch adaptively.

  Since the process of the entire target image correction unit 54a is a simple process that does not require a macroblock extraction process or a reconstruction process, an increase in processing load can be suppressed. Conversely, the process of the target image partial correction unit 54b is superior to the process of the entire target image correction unit 54a in terms of reducing high-frequency noise and improving compression efficiency. However, it is inferior in terms of an increase in processing load. That is, both processes are in a trade-off relationship.

  For example, the switching unit 55 can detect the processing load of the image processing device 200 with reference to the occupation amount of the buffer 90, the bandwidth margin beat rate of the compression-encoded image data, and the like. Those detected values are compared with a predetermined third threshold value, and if the detected value exceeds that value, that is, if the processing load is large, the entire target image correcting unit 54a is selected. Conversely, when the detected value is equal to or smaller than the third threshold, that is, when the processing load is small, the target image partial correction unit 54b is selected.

  In addition, the switching unit 55 can detect the degree of correlation between frame images included in a moving image with reference to the following values. That is, a change value of a quantization scale that is variably controlled, a change value of a motion vector between frame images, or a difference in code amount between adjacent frame images can be used as the value. Those values are compared with a predetermined fourth threshold value, and when the value exceeds that value, that is, when the processing load is large, the entire target image correction unit 54a is selected. Conversely, when the detected value is equal to or smaller than the fourth threshold, that is, when the processing load is small, the target image partial correction unit 54b is selected.

  The third threshold value and the fourth threshold value can be set based on experiments and simulations by the designer. Moreover, you may use both determination processing together by AND condition or OR condition.

  As described above, according to the fifth embodiment, the processing load, the correction accuracy, and the compression efficiency can be changed by switching between the processing according to the first embodiment and the processing according to the third embodiment depending on the situation. The relationship can be optimized.

FIG. 14 is a diagram illustrating the configuration of the preprocessing unit 50 according to the sixth embodiment. The preprocessing unit 50 according to the present embodiment is obtained by adding a switching unit 55 to the preprocessing unit 50 according to the second embodiment illustrated in FIG.
The switching unit 55 determines whether the image after correction by the reference image correction unit 51 or the image before correction is used as a reference image to be supplied to the inter-frame prediction encoding unit 52. Switching is performed based on at least one of the load and the degree of correlation between frame images included in a moving image to be processed by the apparatus. You may switch adaptively. The specific consideration of the switching unit 55 applies to the description of the fifth embodiment.

  As described above, when the image corrected by the reference image correction unit 51 is used as a reference image, it is superior in terms of correction accuracy and compression efficiency than in the case of using an image before correction, but inferior in terms of processing load. To do. That is, both processes are in a trade-off relationship.

  As described above, according to the sixth embodiment, the processing load and the correction accuracy are changed by switching between using the image corrected by the reference image correcting unit 51 as the reference image or using the uncorrected image according to the situation. The compression efficiency relationship can be optimized.

FIG. 15 is a diagram illustrating the configuration of the preprocessing unit 50 according to the seventh embodiment. The preprocessing unit 50 according to the present embodiment is obtained by adding a switching unit 55 to the preprocessing unit 50 according to the fourth embodiment illustrated in FIG.
The switching unit 55 determines whether to use the image corrected by the reference image correcting unit 51 or the image before correction as the reference image to be supplied to the target image correcting unit 54, and whether to use the image before correction or not. Switching is performed based on at least one of the correlation degrees between the frame images included in the moving image to be processed by the apparatus. You may switch adaptively. The specific consideration of the switching unit 55 applies to the description of the fifth embodiment.

  As described above, when the image corrected by the reference image correction unit 51 is used as a reference image, it is superior in terms of correction accuracy and compression efficiency than in the case of using an image before correction, but inferior in terms of processing load. To do. That is, both processes are in a trade-off relationship.

  As described above, according to the seventh embodiment, the processing load and the correction accuracy are changed by switching between using the image corrected by the reference image correcting unit 51 as the reference image or using the uncorrected image according to the situation. The compression efficiency relationship can be optimized.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  The switching unit 55 performs the processing of the basic example, the processing of the first modification, and the processing of the second modification in the third embodiment between the processing load of the image processing apparatus 200 and the frame images included in the moving image to be processed by the apparatus. Switching may be performed based on at least one of the correlation degrees. As a result, the relationship among processing load, correction accuracy, and compression efficiency can be optimized.

  The reference image correction unit 51 has a strong correlation with the encoded image (Fp2) temporally preceding the encoded image (Fp3) in the encoded image (Fp3) temporally preceding the target image. The area of the part was corrected. In this regard, in the encoded image (Fp2) temporally preceding the encoded image (Fp3), a part having a strong correlation with the encoded image (Fp1) temporally preceding the encoded image (Fp2) These areas may be corrected. Furthermore, you may correct | amend retroactively. Furthermore, high frequency noise can be reduced.

  When the reference image correction process is performed, the motion search process that is subsequently executed by the target image correction unit 54 may be executed only for the blocks extracted by the reference image correction process. In this case, the calculation amount can be reduced.

It is a figure which shows the structure of the imaging device carrying the image processing apparatus which concerns on embodiment. FIG. 3 is a diagram illustrating a configuration of a preprocessing unit according to the first embodiment. FIG. 6 is a diagram illustrating a configuration of a preprocessing unit according to a second embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a third embodiment. It is a figure which shows the mode of the correction | amendment image generation concerning Example 3. FIG. 10 is a flowchart illustrating a procedure of image processing according to the third embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a first modification of the third embodiment. 10 is a flowchart illustrating a procedure of image processing according to a first modification of the third embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a second modification of the third embodiment. 10 is a flowchart illustrating a procedure of image processing according to a second modification of the third embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a fourth embodiment. It is a figure which shows the mode of the correction | amendment image generation concerning Example 4. FIG. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a fifth embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a sixth embodiment. FIG. 10 is a diagram illustrating a configuration of a preprocessing unit according to a seventh embodiment.

Explanation of symbols

  20 frame image segmentation unit, 30 orthogonal transform unit, 40 quantization unit, 50 preprocessing unit, 51 reference image correction unit, 52 interframe prediction encoding unit, 53 decoding unit, 54 target image correction unit, 54a overall target image correction 54b target image partial correction unit, 541 motion compensation unit, 542 block extraction unit, 543 inter-frame prediction encoding unit, 544 decoding unit, 545 intermediate image generation unit, 546 image assembly unit, 55 switching unit, 60 intra-frame prediction An encoding unit, an inter-frame prediction encoding unit, an 80 variable length encoding unit, a 90 buffer, a 100 imaging unit, a 200 image processing device, and a 300 imaging device.

Claims (11)

An image processing device for encoding a moving image,
A pre-processing unit that performs pre-processing for taking over part of the image quality of an encoded image that has been temporally preceded with respect to a target image to be subjected to intra-frame prediction encoding,
An intra-frame prediction encoding unit that performs intra-frame prediction encoding on the target image subjected to the pre-processing by the pre-processing unit;
An image processing apparatus comprising: The pre-processing unit is
An inter-frame prediction encoding unit that performs inter-frame prediction encoding of the target image using a decoded image of the encoded image as a reference image;
A decoding unit that decodes the encoded image encoded by the inter-frame prediction encoding unit and passes the decoded image to the intra-frame prediction encoding unit;
The image processing apparatus according to claim 1, further comprising: The pre-processing unit is
In the encoded image that temporally precedes the target image, a partial region having a strong correlation with the encoded image that precedes the encoded image is corrected, and the corrected image is used as the reference image as the frame. The image processing apparatus according to claim 2, further comprising a reference image correction unit that is passed to the inter prediction encoding unit. The pre-processing unit is
2. The image processing apparatus according to claim 1, further comprising: a target image correction unit that corrects a part of the target image having a strong correlation with the encoded image and passes the corrected image to the intra-frame prediction encoding unit. An image processing apparatus according to 1. The target image correction unit
For each of a plurality of divided blocks in the target image, a motion compensation unit that searches for a motion in the decoded image of the encoded image and obtains a motion vector and a predicted image;
A block extraction unit that extracts a block in which the motion vector is smaller than a first threshold and a difference from the predicted image is smaller than a second threshold among a plurality of divided blocks in the target image;
An intermediate image generation unit that generates an intermediate image between the original image of the block extracted by the block extraction unit and the predicted image corresponding thereto;
An image assembling unit that generates the corrected image by replacing the intermediate image generated by the intermediate image generating unit with an original image corresponding to the intermediate image in the target image;
The image processing apparatus according to claim 4, further comprising: The target image correction unit
For each of a plurality of divided blocks in the target image, a motion compensation unit that searches for a motion in the decoded image of the encoded image and obtains a motion vector and a predicted image;
A block extraction unit that extracts a block in which the motion vector is smaller than a first threshold and a difference from the predicted image is smaller than a second threshold among a plurality of divided blocks in the target image;
An inter-frame prediction encoding unit that performs inter-frame prediction encoding with the decoded image as a reference image for at least the block extracted by the block extraction unit;
A decoding unit for decoding the block encoded by the inter-frame prediction encoding unit;
An intermediate image generation unit that generates an intermediate image between an image of a block extracted by the block extraction unit and decoded by the decoding unit and a corresponding original image;
An image assembling unit that generates the corrected image by replacing the intermediate image generated by the intermediate image generating unit with an original image corresponding to the intermediate image in the target image;
The image processing apparatus according to claim 4, further comprising: The target image correction unit
For each of a plurality of divided blocks in the target image, a motion compensation unit that searches for a motion in the decoded image of the encoded image and obtains a motion vector and a predicted image;
A block extraction unit that extracts a block in which the motion vector is smaller than a first threshold and a difference from the predicted image is smaller than a second threshold among a plurality of divided blocks in the target image;
An inter-frame prediction encoding unit that performs inter-frame prediction encoding with the decoded image as a reference image for at least the block extracted by the block extraction unit;
A decoding unit for decoding a block encoded by the inter-frame prediction encoding;
An image assembling unit that generates the corrected image by replacing the image of the block extracted by the block extracting unit and decoded by the decoding unit with the original image corresponding to the image of the block in the target image;
The image processing apparatus according to claim 4, further comprising: The pre-processing unit is
A reference image correction unit that corrects a part of the encoded image temporally preceding the target image and having a strong correlation with the encoded image temporally preceding the encoded image;
A target image correction unit that corrects a part of the target image that is highly correlated with the corrected image corrected by the reference image correction unit, and passes the corrected image to the intra-frame prediction encoding unit;
The image processing apparatus according to claim 1, further comprising: The pre-processing unit is
The target image is subjected to inter-frame prediction encoding using a decoded image of the encoded image temporally preceding the image as a reference image, the encoded image is decoded, and the entire target image is corrected and passed to the intra-frame prediction encoding unit And
A target image partial correction unit that corrects a part of the target image that has a strong correlation with the encoded image temporally preceding the image and passes the corrected image to the intra-frame prediction encoding unit;
Whether to enable the entire target image correction unit or the target image partial correction unit depends on at least one of the processing load of the image processing apparatus and the correlation between the frame images included in the moving image. A switching unit to switch based on,
The image processing apparatus according to claim 1, further comprising: The pre-processing unit is
Whether the image after correction by the reference image correction unit or the image before correction is used as the reference image, the processing load of the image processing device and the degree of correlation between the frame images included in the moving image, The image processing apparatus according to claim 3, further comprising a switching unit that switches based on at least one of them. An imaging unit for acquiring a moving image;
The image processing device according to any one of claims 1 to 10, which processes a moving image acquired by the imaging unit;
An imaging apparatus comprising:

Images