JP2005122601A - Image processing apparatus, image processing method and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily confirm the effects of image processing in which the picture quality or the like of details of an image is remarkably changed. <P>SOLUTION: A difference between the pixel values of pixels existing on the same position of the same block in images obtained before and after image processing is calculated. When the resolution values of the images obtained before and after the image processing are different from each other, the resolution of the image obtained before the processing is adjusted, so as to be equal to the resolution of the image obtained after the processing and then the difference is calculated. Differences of pixel values in the whole blocks are calculated by using the calculated difference of pixel values in each pixel, the differences are compared among blocks and the block in which the difference is maximum is specified. Images obtained by enlarging a prescribed area in the specified block are respectively generated from the image data of the images obtained before and after the image processing and displayed on the same picture. Consequently an area in which the effects of image processing are remarkable is displayed as an enlarged image and the effects of the image processing can be easily checked. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for confirming the effect of image processing when a desired image processing is performed on a certain image, and in particular, in an image after image processing, an area having the largest image processing effect. The present invention relates to a technology that can be enlarged and displayed.

When image processing such as sharpening, blurring, or noise reduction is performed on a certain image, it may be desired to confirm the effect of the image processing on the processed image. In general, the effect of this image processing appears as a difference in image quality before and after the image processing, so the effect of the image processing can be confirmed by comparing the image quality before and after this image processing. Can do.
Conventionally, in order to confirm a difference in image quality or the like between a plurality of images, there has been a technique of displaying the difference in a graph or displaying it as an image.
As an example of such a technique, one described in Patent Document 1 below is known.

Japanese Patent Laid-Open No. 6-237917

  However, in the case of using an image processing method in which the difference in image quality due to image processing becomes significant in the details of the image, the difference in image quality before and after image processing, that is, the effect of image processing is There was a problem that it was difficult to confirm.

  The present invention has been made in order to solve the above-described problems. Even when image processing is performed such that changes in image quality or the like are noticeable in image details, the effect of the image processing is easily achieved. The purpose is to be able to confirm.

In order to solve at least a part of the above-described problems, an image processing apparatus according to the present invention includes a first still image and a second obtained as a result of performing desired image processing on the first still image. An image processing apparatus capable of displaying a still image,
An area specifying processor that compares the first still image with the second still image and specifies an area in which a change in a predetermined pixel value is maximum before and after the desired image processing;
An enlargement processing unit that obtains a first region enlarged image by enlarging the specific region in the first still image and obtains a second region enlarged image by enlarging the specific region in the second still image; ,
A display processing unit that displays the first region enlarged image and the second region enlarged image in parallel;
It is a summary to provide.

In the image processing apparatus of the present invention, the region specifying processing unit compares the first still image before performing the desired image processing with the second still image obtained as a result of performing the desired image processing. For each pixel at the same position, a change in a predetermined pixel value is grasped, and a predetermined area where the change is maximum is specified.
On the other hand, the enlargement processing unit enlarges the predetermined regions specified by the region specifying processing unit in the first still image and the second still image, respectively, so that the first region expansion image and the second region expansion are performed. The image is generated, and the image processing unit displays the first region enlarged image and the second region enlarged image generated by the enlargement processing unit side by side.

  When the desired image processing is performed on the image, in the processed image, the portion where the effect of the image processing is large is, of course, that the predetermined pixel value of the pixel in the portion changes greatly compared to before the processing. Assuming that the above-described configuration is adopted, it is possible to specify a region where the above-described image processing effect is maximized. In addition, the images before and after the image processing can be compared and the images before and after the image processing can be compared at a glance by enlarging the specified area and displaying the images in parallel. However, even if the image processing is such that the image quality changes, it is easy to confirm the effect of the image processing.

Further, in the image processing apparatus of the present invention, when the first still image and the second still image have different resolutions, the second still image is adjusted by adjusting the resolution of the first still image. It further includes a resolution adjustment unit that equals the resolution of the still image,
It is preferable that the area specifying processing unit compares the first still image with the adjusted resolution and the second still image.

  By adopting such a configuration, even if the first still image and the second still image have different resolutions, the resolution of the first still image becomes equal to the resolution of the second still image. Thus, the pixels are present at the same position in the first still image and the second still image. Therefore, the area specifying processing unit can grasp the change in the pixel value of each pixel at the same position for all the pixels included in the first still image and the second still image, and the predetermined pixel value It is possible to accurately identify the region where the change is maximum.

The image processing apparatus of the present invention further includes a specific area image generation unit that generates a third still image including the specific area from the first still image or the second still image,
The display processing unit may display the third still image and a mark indicating the position of the specific area in the third still image.

  In the present invention, if only a region where the desired image processing effect is maximized is displayed in an enlarged manner, the enlarged and displayed regions are images of the first still image and the second still image. This makes it difficult to check which position it is in. However, with the configuration as described above, it is possible to clearly indicate the position of the above-described enlarged display area in the first still image and the second still image. In addition, in the second still image, it is possible to easily confirm which region has the maximum image processing effect.

  In addition, the region specifying unit divides the first still image and the second still image into a plurality of similar blocks, and the same for the first still image and the second still image. The blocks at the positions are sequentially compared to derive a block having the maximum change in the predetermined pixel value before and after the desired image processing, and the predetermined area is specified based on the block. Good.

  By adopting such a configuration, a region where the effect of desired image processing is maximized is specified based on a change in a predetermined pixel value of a plurality of pixels included in the block. Can be specified.

  The desired image processing may be image processing for generating one relatively high-resolution still image from a plurality of relatively low-resolution still images.

Note that the present invention can be realized in various modes as described below.
(1) Image processing method.
(2) A computer program for realizing the above-described image processing apparatus and image processing method.

  When the present invention is configured as a computer program or a recording medium recording the program, the entire program for controlling the operation of the above-described image processing apparatus may be configured, or only a portion that performs the function of the present invention. It may be configured. In addition, as a recording medium, a printed matter on which a code such as a flexible disk, a CD-ROM, a DVD-ROM / RAM, a magneto-optical disk, an IC card, a memory card, a magnetic tape, a ROM cartridge, a punch card, a barcode is printed, Various computer-readable media such as an internal storage device (a memory such as a RAM or a ROM) and an external storage device can be used.

Hereinafter, the best mode for carrying out the present invention will be described in the following order based on examples.
A. Example:
A1. Device configuration:
A2. Specific behavior:
A3. Detailed operation of sharpening process:
A4. Effects of the embodiment:
B. Modified example

A. Example:
A1. Device configuration:
FIG. 2 is an explanatory diagram showing a schematic configuration of the image processing apparatus 10 according to the embodiment of the present invention. The image processing apparatus 10 includes a computer 20, a keyboard 30, a mouse 31, a video camera 32, and a DVD drive 33 as devices for inputting data to the computer 20, and a display 40 as devices for outputting images and the like. And a printer 41. The video camera 32 can read out still image data and moving image data stored on the magnetic tape 35 to be inserted, in addition to shooting a moving image. The DVD drive 33 can read still image data and moving image data stored in the inserted DVD 34.

  The computer 20 includes a CPU 21, a memory 22, a hard disk 23, and an input / output interface unit 24, and each is connected by an internal bus 25. Here, the input / output interface unit 24 includes a group of interfaces for connecting the above-described keyboard 30, mouse 31, video camera 32, DVD drive 33, display 40, and printer 41 to the computer 20. Part.

In the computer 20, the application program 100 operates under a predetermined operating system. Various drivers are incorporated in this operating system, and the above-described keyboard 30, mouse 31, video camera 32, DVD drive 33, display 40, and printer 41 are controlled. When the application program 100 is activated and loaded into the memory 22, the CPU 21 executes the application program 100, thereby causing the image processing unit 100a, the 1-frame high resolution processing unit 100b, the region specifying processing unit 100c, and the enlargement. It functions as the processing unit 100d and the display processing unit 100e.
The one-frame high resolution processing unit 100b corresponds to the resolution adjustment unit and the specific region image generation unit described in the claims, and the region specification processing unit 100c corresponds to the region specification processing unit described in the claims. The enlargement processing unit 100d corresponds to the enlargement processing unit described in the claims, and the display processing unit 100e corresponds to the display processing unit described in the claims.

A2. Specific behavior:
In this embodiment, after performing desired image processing on the image, as the effect clarification processing, in the image after image processing, an area where the effect of the image processing is large is specified, and the specified area is enlarged and displayed. By doing so, the user can easily confirm the effect of the image processing.
In addition, as an image process performed on an image, a case where a sharpening process for generating a high-resolution and high-quality still image from a moving image is described as an example.
In the present embodiment, when desired image processing is performed on an image, the portion of the processed image that has a large effect on the image processing naturally has pixel data (pixel value) of the pixel of that portion. Assuming that there is a large change compared to before processing, by determining the change in pixel data before and after processing of each pixel, it is possible to identify a region where the effect of image processing in the processed image is large Yes.
Note that YCrCb data, which is information on luminance and color difference, is used as pixel data of each pixel constituting the image. Among the YCrCb data, in particular, regarding the luminance value Y, a change before and after the processing is obtained, An area where the effect of processing is large is specified.

  FIG. 1 is an explanatory diagram illustrating a user interface of the image processing apparatus 10 according to the present exemplary embodiment. 1A is an explanatory diagram showing a movie operation dialog D1 displayed on the display 40. FIG. (B) is an explanatory diagram showing an example of block division of a frame image obtained as a result of the sharpening process. Further, (C) is an explanatory diagram showing a sharpening dialog D2 displayed on the display 40. FIG.

  In FIG. 2, moving image data stored on the magnetic tape 35 inserted into the video camera 32 and moving image data stored on the DVD 34 inserted into the DVD drive 33 are stored in the keyboard 30 and the mouse 31. When designated and read by the operation, a movie operation dialog D1 is displayed on the display 40 as shown in FIG. At this time, the first frame image of the designated moving image data is displayed as a still image in the image display frame W11 in the movie operation dialog D1.

If the above-described designated moving image data exists as a file, the file name is also displayed in the movie operation dialog D1. For example, in FIG. Assuming that a moving image file “avi” is designated, the file name is displayed. In addition, in the movie operation dialog D1, as shown in FIG. 1A, a moving image operation including the above-mentioned designated moving image playback, stop, fast forward, reverse, and pause operation buttons. Menu M1 is displayed.
When the playback operation button m10 in the moving image operation menu M1 is selected by operating the keyboard 30 or the mouse 31, the playback of the moving image starts and a moving image is displayed in the image display frame W11. Further, when the pause operation button m11 in the moving image operation menu M1 is selected during the reproduction of the moving image, the reproduction of the moving image is stopped, and the image is displayed at the timing as shown in FIG. The frame image displayed in the display frame W11 is displayed as a still image.
In the movie dialog D1, a still image operation menu M2 for performing various processes on the displayed frame image is also displayed. The still image operation menu M2 includes a “clear” operation button m20, a “print” operation button m21, and a “save” operation button m22.
In this state, when the “print” operation button m21 is selected by operating the keyboard 30 or the mouse 31 shown in FIG. 2, the image data of the frame image displayed in the image display frame W11 is included in the moving image data. And is sent to the printer 41 shown in FIG. 2 to print a frame image. When the “save” operation button m22 is selected by operating the keyboard 30 or the mouse 31, the image data of the frame image displayed in the image display frame W11 is extracted from the moving image data. Are stored in the hard disk 23 and the DVD 34 shown in FIG.
On the other hand, when the “sharpening” operation button m20 is selected by operating the keyboard 30 or the mouse 31, the sharpening dialog D2 shown in FIG. 1 (C) is displayed on the display 40. In addition, an effect clarification process that is a characteristic part of the present invention is executed.
When the “sharpening” operation button m20 shown in FIG. 1A is selected, a sharpening process is first executed by the image processing unit 100a which is a functional part of the CPU 21 shown in FIG.
In the sharpening process, first, image data of a plurality of frame images including the frame image FA are cut out from the moving image data and stored in the memory 22 shown in FIG. (Note that the extraction from the moving image data and the storage in the memory 22 are hereinafter referred to as “capture”.) Then, one reference frame image among the plurality of captured frame images and other objects A frame image is determined, and interpolation processing is performed using image data of the reference frame image and the target frame image, and a sharpened image FB in which the resolution of the reference frame image is increased is generated. Then, the image data of the sharpening processed image FB is stored in the memory 22, and the sharpening processing ends.
Note that the capture of the frame image may be executed within the sharpening process as described above, or may be executed when the pause operation button m11 in the moving image operation menu M1 shown in FIG. 1 is selected. Absent.
The detailed operation of the sharpening process will be described later. Hereinafter, in the sharpening process, it is assumed that the frame image FA is selected as the reference frame image, and the resolution of the sharpening process image FB with respect to the resolution of the frame image FA is set. The magnification is assumed to be magnification A.
Thus, when the sharpening process is completed, an effect clarifying process is executed next.

FIG. 3 is a flowchart showing the processing procedure of the effect clarification processing as one embodiment of the present invention.
In the effect clarification processing, first, in the one-frame high resolution processing (step S3), the resolution of the frame image FA is increased to the same resolution as that of the sharpened processing image FB. This 1-frame resolution enhancement processing (step S3) is executed by the 1-frame resolution enhancement processing unit 100b shown in FIG.

  In the 1-frame high-resolution processing (step S3), interpolation processing is performed using only the image data of the frame image FA stored in the memory 22, and the 1-frame high-resolution having the same resolution as that of the sharpened image FB is obtained. An image FC is obtained. However, as this interpolation processing method, among the bi-linear method, the bi-cubic method, or the nearest neighbor method, the same interpolation processing method as the interpolation processing in the sharpening processing is used, and the same magnification A is used.

For example, in the sharpening process, when the interpolation process is performed using the bi-linear method to obtain a sharpened processed image FB that is 1.5 times higher in resolution than the frame image FA, 1 Similarly, in the frame high-resolution processing (step S3), interpolation processing is performed using the bi-linear method, and the resolution of the frame image FA is increased to 1.5 times higher resolution. Assume that a converted image FC is obtained.
Then, the image data of the 1-frame high-resolution image FC obtained in this way is stored in the memory 22.

Subsequent to the one-frame resolution enhancement processing (step S3), the block luminance difference calculation processing (step S4) and the block luminance difference maximum block identification processing (step S5) are executed by the region identification processing unit 100c shown in FIG. .
In the block luminance difference calculation process (step S4), first, the 1-frame high-resolution image FC and the sharpening-processed image FB are compared, paying attention to the blocks at the same position as the target block, and the same included in those blocks For each pixel at the position, a difference in luminance (hereinafter referred to as “luminance difference”) is calculated. Then, when the luminance difference for all pixels is calculated for the target block, the total value of all the luminance differences (hereinafter referred to as “block luminance difference”) is calculated, and the value is stored in the memory. 22. The above procedure is executed for all blocks by sequentially switching the target blocks.
Subsequently, in the block luminance difference maximum block specifying process (step S5) to be executed, the block luminance difference for each block calculated in the block luminance difference calculating process (step S4) is compared with each other. Then, the block having the largest block luminance difference is specified, and the block number is stored in the memory 22.
The detailed procedure of the block luminance difference calculation process (step S4) and the block luminance difference maximum block specifying process (step S5) will be described later.

  Subsequently, the enlargement processing of the maximum luminance difference block (step S6) is executed by the enlargement processing unit 100d shown in FIG. In this process, first, the block number stored in the memory 22 is referred to in the block luminance difference maximum block specifying process (step S5). Subsequently, based on the referenced block number, the center position of the corresponding block is selected from the image data of the sharpened image FB and the image data of the 1-frame high-resolution image FC stored in the memory 22. Image data of a predetermined area at the center is extracted. Then, an enlargement process is performed based on the extracted image data of the predetermined area, and an area enlarged image Fb and an area enlarged image Fc are generated. Then, the image data of the region enlarged images Fb and Fc are stored in the memory 22. Note that an enlargement ratio in the above-described enlargement process is an enlargement ratio B.

  Note that the enlargement processing at this time is performed by interpolation processing using a bi-linear method, a bi-cubic method, a nearest neighbor method, or the like, but each of the sharpening processing image FB and the one-frame high-resolution image FC is performed. For the predetermined regions, the same interpolation method is used, and the same enlargement ratio B is used.

  Note that the size of the predetermined area described above is defined in advance by the enlargement ratio B and the sizes of image display frames W22 and W23 described later.

  Subsequently, display processing (step S7) is executed by the display processing unit 100e shown in FIG. In this process, the respective image data of the area enlarged images Fb and Fc stored in the memory 22 are read out, and as shown in FIG. 1C, the image display frame W22 in the sharpening dialog D2 is read. , W23 are displayed side by side. FIG. 1C shows that the 20th block shown in FIG. 1B is determined as a result of the block luminance difference maximum block specifying process (step S5).

In addition to the parallel display of the two region enlarged images Fb and Fc described above, a frame image obtained by reducing the frame image FA (hereinafter referred to as “reduced”) as shown in FIG. This is referred to as a “frame image.”) Fa is displayed in the image display frame W24.
Based on the image data of the frame image FA stored in the memory 22, the reduced frame image Fa is obtained from the bi-linear method, the bi-cubic method, or the nearest image by the one-frame high resolution processing unit 100b shown in FIG. An interpolation process using a neighbor method or the like is performed and generated.
Further, a crosshair cursor Cs is displayed on the reduced frame image Fa. The crosshair cursor Cs is displayed at the center position of the block determined in the block luminance difference maximum block specifying process (step S5), and is displayed in parallel. The displayed two area enlarged images Fb and Fc are marks indicating which positions of the block are enlarged images of the predetermined area.

  Note that the reduced frame image Fa is not an image for comparison with other images, but is an image only for indicating to which block the region enlarged image Fb and the region enlarged image Fc correspond. The image quality is not limited, and any image size that can be displayed in the image display frame W24 may be used. Therefore, if the frame image FA or the sharpened image FB has an image size that can be displayed in the image display frame W24, the frame image FA or the sharpened image FB is displayed instead of the reduced frame image Fa. It doesn't matter.

  By the way, as shown in FIG. 1C, an operation menu M3 including a “print” operation button m31, a “save” operation button m32, and a “close” operation button m33 is displayed in the sharpening dialog D2. The Here, when the “print” operation button m31 is selected by operating the keyboard 30 and the mouse 31 shown in FIG. 2, the image data of the sharpened image FB is sent to the printer 41, and the sharpened image FB is displayed. Printed. When the “save” operation button m32 is selected by operating the keyboard 30 or the mouse 31, the image data of the sharpened image FB is transferred from the memory 22 which is the storage destination at this time from the hard disk 23, the DVD 34, or the like. Are copied and stored in a recording medium. When the “close” operation button m33 is selected by operating the keyboard 30 or the mouse 31, the sharpening dialog D2 is closed and the movie operation dialog D1 is activated and stored in the memory 22. All data such as image data and block numbers will be cleared.

  Hereinafter, detailed procedures for the block luminance difference calculation process (step S4) and the block luminance difference maximum block specifying process (step S5) will be described.

FIG. 4 is a flowchart showing a detailed processing procedure of the block luminance difference calculation process (step S4) in the embodiment of the present invention. First, the block number of the block to be processed is first substituted for the variable j indicating the block number of the block to be processed (step S401). Here, both the sharpened image FB and the one-frame high-resolution image FC are divided into the same number of blocks, and the same block numbers are assigned to the blocks at the same positions. In the present embodiment, as shown in FIG. 1B, the sharpened image FB and the one-frame high-resolution image FC are divided into blocks B1, B2, B3,. C1, C2, C3,..., C36 are divided into the same number of 36 blocks, and block numbers are assigned in this order, 1, 2,.
When the processing is performed in the order of the block numbers determined as described above, the block numbers “1” of the blocks B1 and C1 are substituted into the variable j in step S401.

Subsequently, the initial value “0” (zero) is substituted into the variable SV (j) (step S402). The variable SV (j) indicates the block luminance difference of the jth block. As described above, when the first block is selected, the variable SV (1) = 0.
Subsequently, the pixel number to be processed first is substituted for the variable i indicating the pixel number in the block (step S403). This pixel number is, for example, like the block number determined as shown in FIG. 1B, so that the upper left corner pixel in each block is the first, and the right adjacent pixel is the second, Determine in order. Accordingly, “1” is substituted into the variable i as the first pixel number, and the variable i = 1.

Subsequently, a difference in luminance value between pixels having the same pixel number in the sharpened image FB and the one-frame high-resolution image FC is calculated (step S404). This is because the luminance value VFB (i) of the i-th pixel of the j-th block of the sharpened image FB and the luminance value VFC (i) of the i-th pixel of the j-th block of the one-frame high-resolution image FC. ) And the absolute value of the difference is calculated and substituted into the variable ΔV (i).
For example, when the variable j = 1 and the variable i = 1 as described above, the block number 1 is obtained from the image data of the sharpened image FB and the image data of the 1-frame high-resolution image FC, respectively. The luminance value VFB (1) and luminance value VFC (1) of the pixel of pixel number 1 in the block are read out. Then, the absolute value of the difference between the luminance value VFB (1) and the luminance value VFC (1) is calculated, and the result is substituted into the variable ΔV (1).
Subsequently, the value of the variable ΔV (i) obtained in step S404 described above is added to the value of the variable SV (j), and the result is substituted into the variable SV (j) (step S405). As described above, variable SV (1) = 0 (initial value) is set in step S402, and i = 1 is set in step S403. Therefore, as a result of execution of steps S404 and S405, variable SV (j) = Variable ΔV (1).

Subsequently, it is determined whether or not the j-th block has the next pixel to be processed in steps S404 and S405 (step S406). This determination can be made by comparing the number of pixels per block with the variable i. Note that the total number of pixels of the sharpened image FB and the one-frame high-resolution image FC can be calculated from the image size (vertical × horizontal number of pixels), and the number of pixels per block is the total number of pixels and block division. It can be calculated from the number.
As a result of the above determination, if there is a next pixel, the next pixel number is substituted for the variable i (step S407), and thereafter, steps S404 to S407 are executed. Steps S404 to S407 are repeatedly executed until it is determined in step S406 that there is no next pixel. As a result, for example, when processing is performed for the first block with j = 1, assuming that the number of pixels per block is m, finally the variable SV (1) = variable ΔV (1) + variable ΔV (2) +... + Variable ΔV (m). Thus, as the variable SV (1), the total value of the luminance differences for all the pixels included in the block of block number 1, that is, the block luminance difference is obtained.
If it is determined in step S406 that there is no next pixel in the block, the value of the block luminance difference SV (j) at this stage is stored in the memory 22 (step S408).

Subsequently, it is determined whether or not there is a next block for which a block luminance difference is to be calculated (step S409). If there is a next block, the block number of the next block is substituted into the variable j (step S410), and steps S402 to S410 are executed for the block. Steps S402 to S410 are executed until it is determined in step S409 that there is no longer a block whose block luminance difference is to be calculated.
This determination can be made by comparing the number of block divisions (36 in this embodiment) with the value of the variable j.

  When it is determined in step S409 that there is no block for which the block luminance difference is to be calculated, the block luminance difference calculation process (step S4) ends. Finally, as a result of the block luminance difference calculation process (step S4), the values of the luminance differences SV (1) to SV (36) of each block are stored in the memory 22.

  Next, a detailed procedure of the block luminance difference maximum block specifying process (step S5) will be described with reference to FIG.

  FIG. 5 is a flowchart showing a detailed processing procedure of the block luminance difference maximum block specifying process (step S5) in the present embodiment. First, an initial value “−1” is substituted into the variable SVmax (step S501). This variable SVmax is a variable indicating the maximum block luminance difference. Subsequently, the block number to be processed first is substituted for the variable j indicating the block number of the block at the same position in the sharpening processed image FB and the one-frame high-resolution image FC (step S502). . Therefore, when “1” is substituted as the first block number, the variable SVmax = −1 and the variable j = 1.

  Subsequently, the value of the block luminance difference SV (j) of the block with the block number j calculated in the block luminance difference calculation process (step S4) is read from the memory 22 and substituted into the variable SV (j). (Step S503). Subsequently, it is determined whether or not the value of the variable SV (j) is larger than the value of the variable SVmax (step S504). If the value of the variable SV (j) is larger than the value of the variable SVmax, the value of the variable j is substituted for the variable J (step S505). This variable J is a variable indicating the block number of the block having the largest block luminance difference. Subsequent to step S505, the value of the variable SV (j) is substituted into the variable SVmax (step S506), and then the process proceeds to step S507.

  Specifically, the above-described operation after step S505 is assumed to be variable SVmax = −1 and variable j = 1 at the end of step S503 as described above. Here, the value of the block luminance difference SV (1) of the block number “1” is an absolute value as shown in step S404 in FIG. 4. If the value is “3”, the variable SVmax at this time It is determined that the value of the variable SV (1) is greater than the value of the variable SVmax in the determination in step S504. Then, the process proceeds to step S505, where the block number “1” is assigned to the variable J, and in step S506, the value of the variable SV (1) “3” is assigned to the variable SVmax. Therefore, at the end of step S506, the variable SVmax = 3 and the variable J = 1, and the block luminance difference of the first block is the maximum block luminance difference at this stage, and its value is 3.

  On the other hand, if the value of the variable SV (j) is smaller than or equal to the value of the variable SVmax in step S504, the process proceeds to step S507. In this case, the values of the variable J and the variable SVmax do not change. Then, it is determined whether or not there is a next block for executing steps S503 to S506 (step S507). If there is a next block, the block number of the next block is substituted into variable j (step S508), and steps S503 to S508 are executed for the block. Steps S503 to S508 are repeatedly executed until it is determined in step S507 that there is no next block.

If it is determined in step S507 that there is no next block, the value of the variable J at this time is stored in the memory 22 (step S509). At this time, since the block luminance differences SV (1) to SV (36) of all the blocks up to the block number “36” are compared, the value of the variable J is set to the block luminance differences SV (1) to SV (1) to SV # 1. Of the SV (36), this is the block number of the largest block, and the value is stored in the memory 22.
Then, when step S509 is completed, the block luminance difference maximum block specifying process (step S5) ends.

A3. Detailed operation of sharpening process:
The detailed operation of the sharpening process as an example of the image processing in the above-described image processing apparatus will be described below.

FIG. 6 is a flowchart showing the procedure of the sharpening process in the present embodiment. 6A shows the overall procedure of the sharpening process, and FIG. 6B shows the detailed procedure of the synthesis process in FIG. 6A.
The following sharpening process is executed by the image processing unit 100a shown in FIG. 2, and data (such as a deviation amount and a correction amount described later) calculated during the process is stored in the memory 22 shown in FIG. The

  When the sharpening process shown in FIG. 6A is started, a frame image data acquisition process (step S11) is first executed. In this frame image data acquisition process (step S11), first, moving image data is read out from the magnetic tape 35 inserted into the video camera 32 or the DVD 34 inserted into the DVD drive 33 by the image processing apparatus 10, and a moving image data is read out. The image is reproduced. Then, a desired plurality of frame images are captured from the reproduced moving image. In this embodiment, it is assumed that four frame images have been captured.

  Subsequent to the frame image data acquisition process (step S11), a correction amount estimation process (step S12) is executed. In the correction amount estimation process (step S12), first, a frame image serving as a reference for high image quality and high resolution is selected from the captured four frame images. Hereinafter, the selected frame image is referred to as “reference frame image F0”, and the images of other frames are referred to as “target frame images F1 to F3”.

  Next, a shift amount indicating how much each of the three target frame images F1 to F3 is shifted from the reference frame image F0 is sequentially calculated. Note that “displacement” here does not result from the movement of the object to be photographed or the movement of the installation location of the video camera, but the orientation of the video camera, such as camera work called panning or camera shake. This is due to the change only, and it is assumed that all the pixels are shifted by the same amount between different frame images. The deviation is represented by a combination of translational (horizontal and vertical) deviations and rotational deviations.

  When the above-described shift amounts are calculated for the target frame images F1 to F3, correction amounts for correcting the shift are calculated based on the shift amounts. The correction amount is also expressed by a combination of a translational (horizontal and vertical) correction amount and a rotation correction amount, like the shift.

  Subsequent to the correction amount estimation process (step S12), a synthesis process (step S13) is executed. As shown in FIG. 6B, this combining process (step S13) is executed in the order of pixel position correction process (step S131), nearest neighbor pixel determination process (step S132), and pixel interpolation process (step S133). Is done.

  In the pixel position correction process (step S131), the target frame images F1 to F3 are corrected using the correction amount calculated in the correction amount estimation process (step S12). Here, the correction is a pixel position in which the pixel position of each pixel of the target frame images F1 to F3 is moved horizontally by the horizontal correction amount, moved vertically by the vertical correction amount, and rotated by the rotation correction amount. It means to replace with. As a result of this correction processing, the reference frame image F0 and the target frame images F1 to F3 coincide with each other in the overlapping portion.

Next, the nearest pixel determination process (step S132) is executed. FIG. 7 is an explanatory diagram for describing the nearest pixel determination process (step S132) in the present embodiment. In FIG. 7, a part of the matched image of the reference frame image F0 and the target frame images F1 to F3 is enlarged, and the image quality obtained from each of the frame images F0 to F3 and the frame images F0 to F3 is improved. The positional relationship of each pixel of the sharpened image FB is shown.
In FIG. 7, each pixel of the sharpened image FB is indicated by a black circle, and each pixel of the reference frame image F0 is indicated by a white quadrilateral, and the corrected target frame images F1 to F3 are displayed. Each pixel is shown as a quadrilateral with different hatching. In the present embodiment, the sharpened image FB is increased in resolution to a pixel density 1.5 times higher than that of the reference frame image F0. Therefore, as shown in FIG. 7, the distance between the pixels of the sharpened image FB is 2/3 of the distance between the pixels of the reference frame image F0, and the pixel G (k) shown in FIG. As described above, among the pixels of the sharpened image FB, the pixels located every two pixels in the vertical and horizontal directions have the same pixel position as the pixel of the reference frame image F0.

  Now, paying attention to the j-th pixel G (j) in the sharpened image FB, first, this pixel (hereinafter referred to as “target pixel”) G (j) and the target frame image F0. A distance L0 from the pixel closest to the pixel G (j) is calculated. Next, distances L1, L2, and L3 between the pixel closest to the target pixel G (j) and the target pixel G (j) in each of the corrected target frame images F1, F2, and F3 are calculated.

Subsequently, the values of the distances L0 to L3 are compared with each other, and the pixel closest to the target pixel G (j) (hereinafter referred to as “nearest neighbor pixel”) is determined. In this embodiment, as shown in FIG. 7, the pixel of the target frame F3 at the distance L3 is the pixel closest to the target pixel G (j). It will be determined as the nearest neighbor pixel. Hereinafter, the nearest pixel to the pixel G (j) is referred to as the nearest pixel g (F3, i), assuming that it is the i-th pixel of the corrected target frame image F3.
Then, the processing described above is sequentially executed for all the pixels in the sharpening processed image FB, and the nearest pixel is determined for each pixel.

Next, pixel interpolation processing (step S133) is executed. In the pixel interpolation process (step S133), pixel data of each pixel in the sharpened image FB is interpolated from pixel data of other pixels.
In this embodiment, the pixel data used in this interpolation processing is the pixel data of the nearest neighbor pixel determined in the nearest neighbor pixel determination processing (step S132) and the pixel in the corrected frame image to which this nearest neighbor pixel belongs. Pixel data is used.

FIG. 8 is an explanatory diagram for explaining pixel interpolation processing (step S133) using the bilinear method in the present embodiment.
Specifically, if it is determined by the nearest pixel determination process (step S132) that the nearest pixel for the target pixel G (j) is the i-th pixel of the target frame image F3 after correction. 8, in addition to the nearest pixel g (F3, i), the nearest pixel g (F3, i) and the target pixel G (j) are surrounded by the pixel g (F3, j) and the pixel g. The pixel data of the target pixel G (j) is obtained by the bilinear method using the pixel data of (F3, k) and the pixel g (F3, l). In addition to the bi-linear method, various methods such as a bi-cubic method or a nearest neighbor method can be used as the interpolation processing method.

  Then, the image data of the sharpened image FB obtained as a result of the above-described interpolation processing is stored in the memory 22, the pixel interpolation processing (step S133) is ended, and the sharpening processing is also ended.

A4. Effects of the embodiment:
As described above, according to the present embodiment, even when image processing is performed such that the image quality changes in the details of the image as in the above-described sharpening processing, the image is processed in the processed image. A block having the maximum processing effect is specified, and predetermined areas in the specified block are enlarged as area enlarged images Fb and Fc in the sharpening dialog D2, as shown in FIG. This makes it easier to confirm the effect of image processing. In addition, as shown in FIG. 1C, the pre-processed area enlarged image Fb and the post-processed area enlarged image Fc are displayed side by side in the sharpening dialog D2, and at a glance, before and after the process. The area enlarged images Fb and Fc can be compared, and the effect of the image processing can be easily confirmed. Further, by displaying the reduced frame image Fa and the crosshair cursor Cs together with the enlarged region images Fb and Fc in the sharpening dialog D2, which region has the greatest image processing effect in the images before and after the processing. Can be easily confirmed.

  Further, in the one-frame resolution enhancement processing (step S3), the same interpolation processing method used in the sharpening processing and the same magnification A are used to increase the resolution of one frame with the same resolution as the sharpened image FB. By obtaining the image FC, the difference in image quality between the sharpened image FB and the one-frame high-resolution image FC is not caused by a difference in interpolation method or a difference in resolution magnification, but only by the effect of the sharpening process. As a result, the image processing effect can be confirmed accurately.

  Further, in the enlargement process of the luminance difference maximum block (step S6), the enlargement process of the maximum luminance difference block (step S6) can be speeded up by limiting the target range of the enlargement process to a predetermined area instead of the entire block. Further, in the enlargement process of the luminance difference maximum block (step S6), the same interpolation method and the same enlargement are performed when the enlargement process is performed on the respective predetermined regions of the sharpened image FB and the one-frame high-resolution image FC. By using the rate B, the difference in image quality between the region enlarged image Fb and the region enlarged image Fc is not due to the difference in the enlargement rate or the difference in the interpolation method at this time, but only due to the effect of the image processing. can do.

B. Variation:
Note that the present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the scope of the invention. For example, the following modifications are possible. .

(1) In the embodiment of the present invention, the image used for the effect clarification process is the frame image FA and the sharpening process obtained as a result of the sharpening process executed in the image processing apparatus 10 prior to the effect clarification process. Although it is the image FB, it is not limited to this. The sharpening process may be executed in advance by another image processing apparatus, and the resulting image after the sharpening process and the image before the sharpening process may be used in the effect clarifying process. Absent. However, in this case, the images before and after the sharpening process are displayed on a recording medium such as a DVD, a magnetic tape, an MO, a CD-R / RW, or a memory card, or an electric communication medium (not shown). 2 must be stored in the memory 22 shown in FIG.

(2) In the embodiment of the present invention, the case where the sharpening process is performed is described as an example of the image processing. However, the image processing is not limited to the sharpening process. For example, for image processing other than sharpening processing such as blurring or noise reduction, the effect of the image processing can be easily confirmed by using the image processing apparatus 10 of the present invention. In the case of image processing other than the sharpening processing, the corresponding image processing may be executed by the image processing unit shown in FIG. 2 instead of the sharpening processing. In particular, in the case of image processing in which the resolution does not change, it is not necessary to execute the one-frame high resolution processing (step S3) shown in FIG. 3, and accordingly, the block luminance difference calculation processing shown in FIG. In (Step S4), instead of the one-frame high-resolution image FC, the image-processed image (the sharpening-processed image FB in this embodiment) is replaced with the image before the image processing (the frame image FA in this embodiment). ) And the luminance difference of each pixel in each block may be calculated.

(3) In the embodiment of the present invention, the sharpened processed image FB and the one-frame high-resolution image FC are divided into blocks by an appropriately determined number of 36, but instead of this, The size may be divided by determining the number of block divisions so that the size matches the size of a predetermined area in the luminance difference maximum block enlargement process (step S6).
In the present embodiment, depending on the value of the enlargement factor B, the entire image of the block with the largest block luminance difference may not be displayed in the image display frames W22 and W23 in the display process (step S7). In this case, since the luminance difference between the pixels in the non-display area is very large, even if the block is determined to be the block with the largest block luminance difference, the luminance difference of the luminance difference is displayed in the display process (step S7). Since an image of a very large area is not displayed, there may be a problem that it is difficult to confirm the luminance difference, that is, the effect of image processing.
However, as described above, by adopting a block division method in which the size of each block matches the size of the predetermined area, it corresponds to all the pixels included in the block determined to have the largest block luminance difference. Since the enlarged image is displayed, the above-described problem does not occur, and the effect of the image processing can be clearly displayed.

  Furthermore, instead of dividing into blocks and comparing the block luminance differences for each block as described above, areas with high image processing effects are identified by comparing the luminance differences for each pixel instead of dividing into blocks. You may make it do. Specifically, for example, the difference between the luminance values of the pixels at the same position in the sharpening-processed image FB and the one-frame high-resolution image FC is obtained, and the pixel having the largest luminance difference is compared. And the above-mentioned predetermined area may be determined centering on the pixel position.

(4) In the embodiment of the present invention, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software is replaced by hardware. You may do it. For example, the block luminance difference calculation process (step S4) shown in FIG. 3 may be performed by a hardware circuit instead of software (application program 100).

(5) In the embodiment of the present invention, the crosshair cursor Cs on the reduced frame image Fa shown in FIG. 1C indicates to which block the area enlarged images Fb and Fc correspond. However, it may be used to designate a block for which an effect clarification process is desired.
Specifically, after the effect clarification process shown in FIG. 3 is executed, the cross cursor Cs shown in FIG. 1C is continuously placed at the center position of a desired block on the reduced frame image Fa and the keyboard 30 and the mouse. In this state, the block number of a desired block is determined by clicking the mouse 31 or pressing a key such as an enter key on the keyboard 30. Then, the block number determined in this way is used in place of the block number specified in the block luminance difference maximum block specifying process (step S5) shown in FIG. 3, and the luminance difference maximum block enlarging process (step S6), and Display processing (step S7) may be performed.
In this way, when a region of interest is determined in the frame image FA, the effect of image processing on that region can be easily confirmed.

(6) In the embodiment of the present invention, the reduced frame image Fa is displayed as a method of indicating which block the two region enlarged images Fb and Fc displayed in parallel correspond to, and the center of the corresponding block. The position is shown using a cross cursor, but the present invention is not limited to this. In addition to the cross cursor, a cursor such as an arrow may be used to indicate the center position of the corresponding block. Further, the corresponding block may be indicated by bordering and displaying the corresponding block with a red frame or the like. In the present invention, it is sufficient to use a mark that can indicate which position of the block of the enlarged region Fc and the enlarged region Fc displayed in parallel is an image obtained by enlarging a predetermined region of the block.

(7) In the embodiment of the present invention, as shown in FIG. 1C, the enlarged region images Fb and Fc are displayed side by side, but this is not restrictive. For example, a method of displaying them side by side or displaying them in separate dialogs may be used. In the present invention, the region enlarged images Fb and Fc may be displayed on at least the same screen.

(8) In the embodiment of the present invention, in the block luminance difference maximum block specifying process (step S5), the block luminance difference is the same for the blocks at the same position in the sharpened image FB and the one-frame high-resolution image FC. The difference between the pixel data of the pixels at the position is obtained and all of them are calculated, but instead, the same in the blocks at the same position in the sharpened image FB and the one-frame high-resolution image FC. The average value of the pixel data difference of the pixel at the position may be calculated for each block, and the value may be used as the block luminance difference.

(9) In the embodiment of the present invention, the number of block divisions is 36, but the number is not limited to this. Further, both the block number and the pixel number are given natural numbers of 2, 3, 4,... In order with the upper left corner being 1, but this is not restrictive. Each block and each pixel may be assigned numbers that are uniquely determined in any order, and the same position block and the same in the sharpened image FB and the one-frame high-resolution image FC It suffices if the same number is assigned to the pixel at the position.

(10) In the embodiment of the present invention, the luminance value Y is used as the pixel data for obtaining the change before and after the processing. Instead of the luminance value Y, Cr (color difference value between red and luminance) or Cb (blue and luminance). Color difference value) may be used. Also, YCrCb data is used as the pixel data. The YCrCb data is converted into RGB data composed of gradation values of R (red), G (green), and B (blue) using a predetermined conversion formula. Alternatively, other color system data may be used.

(11) In the embodiment of the present invention, the image processing apparatus 10 uses the magnetic tape 35 inserted into the video camera 32 or the DVD 34 inserted into the DVD drive 33 as a recording medium for still image data or moving image data. However, it is not limited to these. Other recording media such as MO, CD-R / RW, or memory card can also be used.

(12) In the embodiment of the present invention, the predetermined area of the block having the largest image processing effect is enlarged and displayed. Instead, the predetermined area of the block having the smallest image processing effect is enlarged and displayed. It is also possible to do. In this case, in step S504 shown in FIG. 5, instead of the value of variable SVmax, the value of variable SVmin indicating the minimum block luminance difference is compared with the value of variable SV (j). Good. In step S504, when the value of the variable SV (j) is smaller than the value of the variable SVmin, the process proceeds to step S505. On the other hand, the value of the variable SV (j) is larger than the value of the variable SVmin. Or if they are equal, the process may proceed to step S507. Furthermore, in step S506, the value of the variable SV (j) may be substituted for the variable SVmin.
By doing in this way, it is possible to easily confirm the region having the smallest image processing effect, and it is possible to make a determination such that additional image processing is performed for the region.
(13) In the embodiment of the present invention, the image data of a plurality of frame images is cut out from the moving image data in the above-described sharpening process. For example, the image data is simply cut out from a plurality of image data arranged in time series. You may do it. As such a plurality of image data arranged in chronological order, a plurality of image data taken continuously by a digital camera or the like can be considered.

Explanatory drawing which shows the user interface of the image processing apparatus 10 in a present Example. 1 is an explanatory diagram illustrating a schematic configuration of an image processing apparatus 10 according to an embodiment of the present invention. The flowchart which shows the process sequence of the effect clarification process as one Example of this invention. The flowchart which shows the detailed process sequence of the block brightness | luminance difference calculation process (step S4) in the Example of this invention. The flowchart which shows the detailed process sequence of the block brightness | luminance difference largest block specific process (step S5) in a present Example. The flowchart which shows the procedure of the sharpening process in a present Example. Explanatory drawing for demonstrating the nearest pixel determination process (step S132) in a present Example. Explanatory drawing for demonstrating the pixel interpolation process (step S133) using the bilinear method in a present Example.

Explanation of symbols

10 ... Image processing device 20 ... Computer 21 ... CPU
22 ... Memory 23 ... Hard disk 24 ... Input / output interface 25 ... Internal bus 30 ... Keyboard 31 ... Mouse 32 ... Video camera 33 ... DVD drive 34 ... DVD
35 ... Magnetic tape 40 ... Display 41 ... Printer D1 ... Movie operation dialog D2 ... Sharpening dialog M1 ... Moving image operation menu M2 ... Still image operation menu M3 ... Operation menu m10 ... Playback operation button m11 ... Pause operation button m20 ... "Clear" operation button m21, m31 ... "Print" operation button m22, m32 ... "Save" operation button m33 ... "Close" operation button W11, W22, W23, W24 ... Image display frame Cs ... Crosshair cursor

Claims (16)

An image processing apparatus capable of displaying a first still image and a second still image obtained as a result of performing desired image processing on the first still image,
An area specifying processor that compares the first still image with the second still image and specifies an area in which a change in a predetermined pixel value is maximum before and after the desired image processing;
An enlargement processing unit that obtains a first region enlarged image by enlarging the specific region in the first still image and obtains a second region enlarged image by enlarging the specific region in the second still image; ,
A display processing unit that displays the first region enlarged image and the second region enlarged image in parallel;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
When the resolutions of the first still image and the second still image are different, a resolution adjustment unit that adjusts the resolution of the first still image to be equal to the resolution of the second still image And further comprising
The area specifying processing unit compares the first still image whose resolution has been adjusted with the second still image, and an image processing apparatus characterized in that the region specifying processing unit compares the second still image with the resolution. The image processing apparatus according to claim 1, wherein:
A specific area image generating unit that generates a third still image including the specific area from the first still image or the second still image;
The display processing unit displays the third still image and a mark indicating the position of the specific region in the third still image. The image processing apparatus according to any one of claims 1 to 3,
The region specifying unit divides the first still image and the second still image into a plurality of similar blocks, and the first still image and the second still image are at the same position. An image characterized by sequentially comparing the certain blocks, deriving a block having a maximum change in a predetermined pixel value before and after the desired image processing, and specifying the predetermined region based on the block Processing equipment. An image processing apparatus according to any one of claims 1 to 4,
The desired image processing is image processing for generating one relatively high-resolution still image from a plurality of relatively low-resolution still images. An image processing method for displaying a first still image and a second still image obtained as a result of performing desired image processing on the first still image,
(A) comparing the first still image with the second still image and identifying a region where the change in the predetermined pixel value is maximum before and after the desired image processing;
(B) a step of enlarging the specific area in the first still image to obtain a first area enlarged image, and enlarging the specific area in the second still image to obtain a second area enlarged image When,
(C) displaying the first region enlarged image and the second region enlarged image in parallel;
An image processing method comprising: The image processing method according to claim 6,
When the resolution is different between the first still image and the second still image,
(D) further comprising adjusting the resolution of the first still image to be equal to the resolution of the second still image;
In the step (a), the first still image whose resolution has been adjusted and the second image are compared. The image processing method according to claim 6, wherein:
(E) generating a third still image including the specific region from the first still image or the second still image;
(F) displaying the third still image and a mark indicating the position of the specific region in the third still image;
An image processing method, further comprising: The image processing apparatus according to any one of claims 6 to 8,
In the step (a), the first still image and the second still image are each divided into a plurality of similar blocks, and the same position is set for the first still image and the second still image. The blocks are sequentially compared with each other, the block having the largest change in the predetermined pixel value before and after the desired image processing is derived, and the predetermined region is specified based on the block. Image processing method. The image processing method according to any one of claims 6 to 9,
The desired image processing is image processing for generating one relatively high-resolution still image from a plurality of relatively low-resolution still images. A program for displaying a first still image and a second still image obtained as a result of performing desired image processing on the first still image,
A first function that compares the first still image with the second still image and identifies a region in which a change in a predetermined pixel value is maximum before and after the desired image processing;
Second function of obtaining a first region enlarged image by enlarging the specific region in the first still image and obtaining a second region enlarged image by enlarging the specific region in the second still image When,
A third function for displaying the first region enlarged image and the second region enlarged image in parallel;
A program to make a computer realize. The program according to claim 11,
When the resolution is different between the first still image and the second still image,
A fourth function for adjusting the resolution of the first still image to be equal to the resolution of the second still image is further realized by a computer, and the first function whose resolution is adjusted is used as the first function. A program for comparing the still image with the second still image. In the program according to claim 11 or 12,
A fifth function for generating a third still image including the specific area from the first still image or the second still image;
A sixth function for displaying the third still image and a mark indicating the position of the specific region in the third still image;
A program that allows a computer to realize this. The program according to any one of claims 11 to 13,
The first function divides the first still image and the second still image into a plurality of similar blocks, and the same position for the first still image and the second still image. Including a function of sequentially comparing the blocks in order to derive a block having a maximum change in a predetermined pixel value before and after the desired image processing and specifying the predetermined area based on the block. A featured program. The program according to any one of claims 11 to 14,
The desired image processing is an image processing for generating one relatively high resolution still image from a plurality of relatively low resolution still images.   A computer-readable recording medium on which the program according to claim 11 is recorded.

Images