JP2007318259A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing method which facilitate to set coding quality for obtaining image quality desired by a user. <P>SOLUTION: The degree of deterioration of image data after coding to the image data before coding is calculated. Information indicating the calculated degree of deterioration is displayed together with the image data before coding or the image data after coding. If a change occurs in the coding setting, such as an image quality adjusting instruction, the degree of deterioration corresponding to setting after the change is calculated and the calculated degree is reflected on the display. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method for encoding an image.

  2. Description of the Related Art Conventionally, a digital video camera is well known as a camera-integrated moving image recording apparatus that captures a subject and compresses and records a moving image obtained thereby. In recent years, the recording medium has changed from a conventional magnetic tape to a disk medium or a semiconductor memory having convenience such as random accessibility. However, since disk media and semiconductor memories generally have a smaller storage capacity than magnetic tape, it is necessary to further increase the efficiency of compression encoding in order to achieve the same recording time as before.

  In recent years, digital video cameras that handle high-definition (HD) video with a larger amount of information have been developed. From this point of view, it is desired to realize more efficient compression coding. Currently, various compression encoding schemes are used, but MPEG is standardly used as a highly efficient compression encoding scheme.

Further, in a video camera that performs compression encoding, it is general that a user can set a recording image quality (see, for example, Patent Document 1).
A high-efficiency encoding method such as the MPEG method is generally an irreversible compression method, and an encoded image is deteriorated with respect to an original image, and the degree of deterioration is not known until playback. For this reason, when photographing an important scene, photographing is performed in advance to check the degree of deterioration of the image, and the recording image quality (for example, bit rate and compression rate) at the time of actual photographing is set.

JP 2001-211141 A

  However, in general, the display unit of a video camera has a smaller screen size and resolution than a TV monitor, and it is impossible to confirm the degree of detail image quality degradation. For this reason, it is difficult to set an optimum recording image quality, and it may be found that an image was shot with an unintended image quality when viewed on a TV monitor after shooting.

  The present invention has been made in view of such problems of the prior art, and an object thereof is to provide an image processing apparatus and an image processing method that make it easy to set encoding quality for obtaining image quality desired by a user. .

  The above-described object is to detect the degree of deterioration of the encoded image data with respect to the input image data, the input means for inputting the image data, the encoding means for irreversibly encoding the image data input by the input means. Image processing according to the present invention, comprising: a deterioration detecting unit that performs input, image data that has been input or encoded, and a display unit that displays information indicating the degree of deterioration detected by the deterioration detecting unit on a display device. Achieved by the device.

  In addition, the above-described objects are an input step for inputting image data, an encoding step for irreversibly encoding image data input by the input means, and a degree of deterioration of the encoded image data with respect to the input image data. And a display step of displaying on the display device the input image data or encoded image data, and information indicating the degree of deterioration detected by the deterioration detection means. This is also achieved by an image processing method.

  With such a configuration, according to the present invention, it is possible to confirm the degree of image quality degradation due to encoding with an objective index, so that the encoding quality for obtaining the image quality desired by the user is easily set. It becomes possible.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a configuration of a moving image recording apparatus as an example of an image processing apparatus according to the first embodiment of the present invention.

  In FIG. 1, an image pickup unit 101 includes a lens, a drive circuit thereof, an image pickup device, and the like, and outputs a subject image as digital data. The moving image recording apparatus also includes a motion compensation prediction circuit 102, a subtractor 103, a DCT (orthogonal transform) circuit 104, a quantization circuit 105, a variable length encoding circuit 106, a recording unit 107, a recording medium 109, and a code amount control circuit 110. Is provided. Furthermore, in addition to the inverse quantization circuit 111, the IDCT (inverse orthogonal transform) circuit 112, the adder 113, the deterioration detection circuit 115, the display image generation circuit 116, the display unit 117, the frame memory 118, and the parameter setting unit 119, a switch 108 and 114. The operations of the components including the switches 108 and 114 and the code amount control circuit 110 are controlled by the system controller 120. The system controller 120 is a microcomputer, for example, and controls operation of the entire apparatus by executing a control program. Furthermore, the system controller 120 also controls the operation of the entire apparatus according to instructions given from the user via the operation unit 122 as necessary.

  The moving image data obtained by imaging by the imaging unit 101 is sequentially stored in the frame memory 118 in the order of the first frame, the second frame, the third frame,. For example, image data is extracted from the frame memory in the order determined according to the encoding method, such as the third frame, the first frame, the second frame,. In this embodiment, an MPEG encoding method is used. In the MPEG system, encoding is performed using a plurality of encoding methods. Specifically, “intra coding” for coding using only image data in the same frame and “inter coding” for predictive coding using the correlation with other frames are used. In addition, inter-coding includes forward prediction coding that can use only temporally past frames and bidirectional prediction coding that can also use temporally future frames as other frames used for inter-frame prediction. is there. A frame that is intra-coded is called an I picture, a frame that is forward-predicted is called a P picture, and a frame that is bi-directionally predictive-coded is called a B picture. Then, in order to realize bi-predictive encoding, encoding is performed in an order different from the input order.

Hereinafter, the encoding operation will be described for each picture type.
In the case of an I picture, data of 16 horizontal pixels and 16 vertical pixels constituting a macroblock (hereinafter referred to as “MB”) which is a coding unit is read from the frame memory 118 and input to the DCT circuit 104. In the case of an I picture, since motion compensation prediction is not performed, zero is output from the motion compensation prediction circuit 102. Therefore, the MB data read from the frame memory 118 is directly input to the DCT circuit 104 without changing its value in the adder 103.

  The MB data after being orthogonally transformed by the DCT circuit 104 is quantized by the quantization circuit 105 with the quantization coefficient output from the code amount control circuit 110. The quantized MB data is input to the inverse quantization circuit 111 and the variable length coding circuit 106, respectively. The variable length coding circuit 106 performs variable length coding on the quantized MB data and outputs it, and at the same time notifies the code amount control circuit 110 of the generated code amount. The code amount control circuit 110 receives the generated code amount received from the variable length coding circuit 106, determines a quantization coefficient to be applied to the next MB, and outputs it to the quantization circuit 105.

  The inverse quantization circuit 111 performs inverse quantization on the image data output from the quantization circuit 105, restores the image data before quantization (image data after orthogonal transformation), and supplies the image data to the IDCT circuit 112. The IDCT circuit 112 performs inverse orthogonal transform on the image data from the inverse quantization circuit 111 to restore the image data before the orthogonal transform (image data read from the frame memory 118). This restored image data is image data that becomes a reference image for predictive encoding to be performed later, and is referred to as a locally decoded image or a local decoded image.

  As described above, when the I picture is encoded, the output of the motion compensation prediction circuit 102 is zero. Therefore, the output of the adder 113 is the same as the local decoded image data output from the IDCT circuit 112. At this time, the switch 114 is controlled to be on, and the local decoded image is stored in the frame memory 118.

  In the case of a P picture, the data of the encoding target MB is input from the frame memory 118 to the motion compensation prediction circuit 102. The motion compensation prediction circuit 102 reads a reference image already stored in the frame memory 118, detects a block (reference block) having a small difference from the encoding target MB, and outputs image data of the detected block. . The subtracter 103 subtracts the detected reference block image data output from the motion compensation prediction circuit 102 from the image data of the encoding target MB supplied from the frame memory 118 and supplies the subtraction result to the DCT circuit 104. To do.

  The DCT circuit 104 orthogonally transforms the subtraction result and supplies it to the quantization circuit 105. The quantization circuit 105 quantizes the difference data after orthogonal transformation output from the DCT circuit 104 and outputs the quantized difference data to the inverse quantization circuit 111 and the variable length coding circuit 106, respectively.

  The inverse quantization circuit 111 inversely quantizes the difference data after quantization, restores the difference data before quantization (difference data after orthogonal transformation), and supplies it to the IDCT circuit 112. The IDCT circuit 112 performs inverse orthogonal transformation on the difference data from the inverse quantization circuit 111 to restore the difference data before the orthogonal transformation. The restored difference data is added to the image data of the reference block by the adder 113 to generate a local decoded image. At this time, the switch 114 is controlled to be in an on state, and the local decoded image is stored in the frame memory 118.

  In the case of a B picture, encoding is performed in the same manner as in the case of the P picture described above. However, since the B picture is not a reference image for prediction, the switch 114 is controlled to be turned off, and the local decoded image data is It is not stored in the frame memory 118. For this reason, it is not necessary to generate a local decoded image, and originally, processing after inverse quantization is not necessary. However, in this embodiment, decoding after inverse quantization is performed even for a B picture in order to detect deterioration described later. Processing (local decoding processing) is also performed.

  Regardless of the picture type, the output of the adder 113 is the same as the image data output during playback. MB data (pre-encoding image) supplied from the frame memory 118 to the subtractor 103 and MB data (reproduced image) supplied from the adder 113 to the frame memory 118 are input to the deterioration detection circuit 115. Then, the deterioration detection circuit 115 calculates the degree of deterioration of the reproduced image with respect to the pre-encoding image. For example, PSNR (Peak Signal-to-Noise Ratio) shown in the following Equation 1 can be used as the degree of deterioration.

In Equation 1, N is the number of MB data, and N = 256 in the case of a luminance signal. Ci represents each pixel value of the pre-encoding image, and Di represents each pixel value of the reproduced image. PSNR [db] is the reciprocal of the mean square error between the pre-encoding image and the post-encoding image, and indicates a smaller value as the difference (deterioration) in the image increases.

  When the user operates the operation unit 122 and the moving image recording apparatus is set to the recording mode, the system controller 120 detects it and controls the switch 108 to be turned off. Then, the above-described encoding process is executed even if there is no instruction to start recording. That is, the encoding process is performed in the same manner as in the normal recording operation, but the recording on the recording medium 109 is not performed.

  As described above, the deterioration detection circuit 115 calculates the PSNR for each MB, and the calculated PSNR value is temporarily stored in a buffer (not shown) in the deterioration detection circuit 115. The display image generation circuit 116 generates a display image to be displayed on the display unit 117 such as an LCD. The image signal picked up by the image pickup unit 101 is held in the frame memory 118 until encoding is performed as described above. However, since the PSNR value is overlaid, the image signal is displayed even after encoding. Hold up.

  When the PSNR values for all MBs of the display image are calculated, the display image generation circuit 116 reads the image signal to be displayed from the frame memory 118. Further, the PSNR value corresponding to the display image is read from the deterioration detection circuit 115, and a composite image in which the display image and a pattern corresponding to the PSNR value are combined for each MB is generated and displayed on the display unit 117. The pattern may be any pattern such as hatching or shading, but it is preferable to avoid a pattern in which the display image is completely invisible.

  FIG. 3 is a diagram for explaining frame management operations of the frame memory 118 and the PSNR buffer in the deterioration detection circuit 115 from when an image signal is input to when it is encoded and output to the monitor. .

  In FIG. 3, F0, F1,... Are image frames, and “I”, “P”, and “B” written on the input image frame are pictures when the input image frame is encoded. Indicates the type and represents an I picture, a P picture, and a B picture, respectively. The frame memory 118 can store a plurality of image frames, and an image frame surrounded by a thick frame represents an encoded frame. A frame with a dash, such as F2 ', represents a local decoded image corresponding to a frame without a dash. FIG. 3 shows an example of the frame management operation. By such frame management, an overlay display of the input image and the PSNR value is possible.

  The degradation detection circuit 115 calculates the PSNR value for each MB of the image frame and also calculates the sum of the PSNR values in the entire frame. Then, the display image generation circuit 116 synthesizes a numerical value representing the image quality into the display image using, for example, a character generator according to the total value of the PSNR values in the entire frame.

  An example of the composite image 202 displayed on the display unit 117 is shown in FIG. The composite image 202 is superimposed on a normal monitor display (display when the display unit 117 is functioning as an electronic viewfinder), and a pattern corresponding to the PSNR value is superimposed and displayed for each MB. For example, the higher the hatching density, the lower the PSNR and the greater the image quality degradation. Further, the character display 204 shows the total PSNR value of the entire frame.

  The user can adjust the image quality, that is, the encoding parameter by operating an image quality setting button or the like included in the operation unit 122 while viewing the composite image 202. That is, the system controller 120 detects an image quality adjustment instruction input and provides the input content to the parameter setting unit 119. The parameter setting unit 119 changes a predetermined encoding parameter according to the given adjustment content, and outputs the changed parameter to each unit that uses the changed parameter.

  There are various encoding parameters. Here, for example, the bit rate of the encoded data is used. For example, when the user gives an instruction to increase the image quality, the parameter setting unit 119 instructs the code amount control circuit 110 to increase the bit rate by a predetermined amount, or takes a predetermined value. To instruct.

  The code amount control circuit 110 controls the bit rate by switching, for example, a quantization table used by the quantization circuit 105 according to an instruction. When the bit rate is increased, the image quality is improved by using a quantization table for performing quantization processing with a smaller quantization width.

  When the start of recording is instructed by a user operation, the system controller 120 controls the switch 108 to be on, and the encoded image data is recorded on the recording medium 109 via the recording circuit 107.

Thus, in this embodiment, the result of the encoding parameter change by the user is immediately reflected in the composite image 202 displayed on the display unit 117. Therefore, the user can set the encoding parameter while confirming the influence of the change of the encoding parameter on the image quality.
In particular, since information indicating the degree of deterioration for each MB is displayed in association with the position of the subject on the display screen, the user can finely adjust the image quality so that the image quality deterioration of the target subject becomes a desired degree. It becomes possible.

  In the composite image 202, the degree of image degradation is presented by objective information such as patterns and numerical values. Therefore, even when the display area of the display unit 117 is small, it is possible to correctly grasp the degree of image quality degradation. .

  In this embodiment, the composite image 202 is generated using the input image. However, by generating the composite image 202 using the encoded image, it is possible to grasp the deterioration of image quality more directly. Also good. In that case, in order to store the locally decoded image in the frame memory 118 even in the case of encoding the B picture, the switch 114 is turned on also in the encoding of the B picture, and the frame management is as shown in FIG. Just do it.

  In this embodiment, the PSNR value is stored in the buffer of the deterioration detection circuit 115. However, the PSNR value is stored using a part of the frame memory 118, and the display image generation circuit 116 stores the necessary PSNR value. You may make it read from the frame memory 118. FIG.

  Furthermore, in the present embodiment, the degree of image quality deterioration for each macroblock is superimposed on the display image. However, when the display unit 117 is very small, for example, when the visibility of the display of the degree of deterioration is reduced due to the superimposed display. Alternatively, only the display indicating the degree of deterioration may be performed. Alternatively, the display image and the display indicating the degree of deterioration can be switched to be displayed. Further, only the MBs whose deterioration degree exceeds a predetermined degree may be displayed in a superimposed manner.

(Second Embodiment)
FIG. 5 is a block diagram showing a configuration of a moving image recording apparatus as an example of an image processing apparatus according to the second embodiment of the present invention. Since the remaining amount calculation circuit 501 is added to the moving image recording apparatus shown in FIG. 1 and other configurations are common, redundant description is omitted.

  In this embodiment, the remaining amount calculation circuit 501 is supplied with an output of the remaining recordable capacity of the recording medium 109 from the recording circuit 107, and an output of a predicted bit rate value from the code amount control circuit 110. Accordingly, the remaining amount calculation circuit 501 can estimate the remaining time that can be recorded on the recording medium based on the current recording image quality. The remaining time is output to the display image generation circuit 116, and the remaining time is combined with the display screen by a character generator or the like. FIG. 6 shows a display example of the composite image 202 in the present embodiment. It can be seen that the display of the remaining time 601 is added to the composite image of FIG.

When the user adjusts the image quality and the encoding parameter is changed by the parameter setting circuit 119, the bit rate output from the code amount control circuit 110 changes accordingly. Therefore, the estimated remaining time is also immediately changed and immediately reflected on the display screen.
Since other operations are the same as those of the moving image recording apparatus of the first embodiment, the description thereof is omitted.

  As described above, according to the present embodiment, the recordable time with the currently set image quality is displayed in real time. Therefore, in addition to the effects of the first embodiment, the user can set an optimum image quality while taking into consideration the remaining recordable amount.

(Third embodiment)
FIG. 7 is a block diagram showing a configuration of a moving image recording apparatus as an example of an image processing apparatus according to the third embodiment of the present invention. Except for the point that the output of the deterioration detection circuit 115 is input to the code amount control circuit 110, the configuration is the same as that of the moving image recording apparatus according to the second embodiment, and thus the description of the common parts is omitted.

  As described above, the degradation detection circuit 115 calculates a PSNR value for each MB. In the present embodiment, the PSNR value for each MB is given to the code amount control circuit 110.

  As described above, the code amount control circuit 110 quantizes the generated code amount output from the variable length coding circuit 106 and the target bit rate determined by the parameter setting circuit 119 according to the content of the image quality adjustment instruction by the user. The quantization table used by the quantization circuit is switched.

At this time, in accordance with an image quality adjustment instruction from the user, the PSNR value threshold for determining the MB to be quantized with a smaller quantization width is changed. For example, in response to an instruction to increase image quality, control is performed so that more MBs are quantized with a smaller quantization width by lowering the threshold value of the PSNR value. In response to an instruction to lower the image quality, conversely, by increasing the threshold value, the number of MBs to be quantized with a small quantization width is reduced.
Since other operations are the same as those of the moving image recording apparatus described in the second embodiment, the description thereof is omitted.

  As described above, according to the present embodiment, the rate of increase of the code amount is suppressed for a portion for which sufficient image quality has already been obtained, and quantization with a smaller quantization width is performed for an MB having a large image quality degradation. Therefore, efficient image quality improvement can be performed while suppressing an increase in the code amount.

(Other embodiments)
In the above-described embodiment, the example in which the PSNR value is used as an objective index representing the degree of image quality degradation due to encoding has been described. However, it goes without saying that other indexes can be used as long as the degree of image quality degradation due to encoding can be expressed.
For example, it is also possible to simply use a difference between image data before encoding and image data after decoding.

  The above-described embodiment can also be realized in software by a computer of a system or apparatus (or CPU, MPU, etc.).

  Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto.

  A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code.

Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers.
That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which a computer program for realizing the above-described embodiment is encrypted and stored is distributed to the user, and key information for decrypting is supplied to the user who satisfies a predetermined condition, and the user's computer It is also possible to enable installation on Windows. The key information can be supplied by being downloaded from a homepage via the Internet, for example.

  Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer.

  Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

1 is a block diagram illustrating a configuration example of a moving image recording apparatus as an example of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows an example of the synthesized image displayed with the moving image recording device which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the example of management of the frame memory in the moving image recording device which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating another example of management of the frame memory in the moving image recording device which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the moving image recording device as an example of the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the synthesized image which the moving image recording device which concerns on the 2nd Embodiment of this invention displays. It is a block diagram which shows the structural example of the moving image recording device as an example of the image processing apparatus which concerns on the 3rd Embodiment of this invention.

Claims (8)

Input means for inputting image data;
Encoding means for irreversibly encoding the image data input by the input means;
Deterioration detecting means for detecting a degree of deterioration of the encoded image data with respect to the input image data;
An image processing apparatus comprising: display means for displaying the input image data or the encoded image data and information indicating the degree of deterioration detected by the deterioration detection means on a display device.   The image processing apparatus according to claim 1, wherein the display unit superimposes and displays the information indicating the degree of deterioration on the input image data or image data obtained by decoding the encoded image data.   The deterioration detection unit divides the input image data or encoded image data into a plurality of regions, detects the deterioration degree for each of the plurality of regions, and the display unit detects the input image. 2. The image processing apparatus according to claim 1, wherein information on the degree of deterioration of each area is superimposed and displayed at a position corresponding to each area in the image data obtained by decoding the data or the encoded image data.   The deterioration detecting means includes decoding means for locally decoding the image data encoded by the encoding means, and the deterioration is based on a difference between the locally decoded image data and the image data before encoding. The image processing apparatus according to claim 1, wherein the degree of the image is detected. And recording means for recording the encoded image data on a recording medium;
Estimating means for estimating the remaining time recordable on the recording medium,
In addition to the image data obtained by decoding the input image data or the encoded image data and the information indicating the degree of deterioration detected by the deterioration detecting means, the display means adds the remaining time estimated by the estimating means. The image processing apparatus according to claim 1, wherein:   6. The image according to claim 1, further comprising a code amount control unit that controls a data amount output from the encoding unit in response to an image quality adjustment instruction by a user. Processing equipment.   The code amount control means acquires the degree of deterioration from the deterioration detection means, and controls the data amount according to whether or not the degree of deterioration is in accordance with the image quality adjustment instruction. The image processing apparatus described. An input step for inputting image data;
An encoding step of irreversibly encoding the image data input by the input means;
A deterioration detecting step of detecting a degree of deterioration of the encoded image data with respect to the input image data;
An image processing method comprising: a display step of displaying the input image data or the encoded image data and information indicating a degree of deterioration detected by the deterioration detection unit on a display device.