JP2006202168A - Generation of high-resolution image using a plurality of low resolution images - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique enabling reduction of a processing load for generating a high-resolution image, using a plurality of low resolution images. <P>SOLUTION: First, using a sheet of a reference low-resolution image among the plurality of input low-resolution images, a pixel value of an output high-resolution image is estimated. Further, using the reference low-resolution image, a low-resolution sheet of an estimated low-resolution image is generated. Also, a pixel value of the high-resolution error image representing an image synthesized from the difference between the above estimated low-resolution image and each input low-resolution image is calculated. Then, by means of negative feedback of the pixel value of the above high-resolution error image, the pixel value of the estimated output high-resolution image is updated. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for generating an image having a high pixel density from a plurality of images having a low pixel density.

  A moving image shot by a digital video camera or the like is also used to generate a still image that represents one scene of the moving image. The moving image includes a plurality of still images (also referred to as frame images). Here, a process of generating a still image having a higher pixel density than the frame image using a plurality of frame images (also referred to as a resolution enhancement process) is used (see, for example, Patent Document 1).

Japanese Patent Laid-Open No. 2004-272751

  However, the use of a plurality of frame images increases the amount of data used for the resolution enhancement processing, and thus the load of the resolution enhancement processing tends to increase. As a result, in order to execute such high resolution processing, there has been a tendency to require high calculation capability such as a high calculation processing speed and a large capacity memory.

Such a problem is not limited to a case where a still image is generated using a moving image, but is a problem common to a case where a still image having a higher pixel density is generated using a plurality of independent still images. .

  The present invention has been made to solve the above-described problems, and provides a technique capable of reducing the processing load of generating a high-resolution image using a plurality of low-resolution images. Objective.

  In order to solve at least a part of the above-described problems, an image processing apparatus according to the present invention provides a high-resolution output high-resolution image having a relatively high pixel density from a plurality of low-resolution input low-resolution images having a relatively low pixel density. An image processing apparatus for generating an estimated output high-resolution image that is an estimated image of the output high-resolution image using a reference low-resolution image that is one of the plurality of input low-resolution images. A high-resolution image estimation unit that calculates pixel values; a low-resolution image estimation unit that creates one low-resolution estimated low-resolution image using the reference low-resolution image; and a relative relationship between the plurality of input low-resolution images Update the pixel value of the estimated output high resolution image using a motion amount calculation unit that calculates a typical motion amount, the pixel value of the plurality of input low resolution images, and the pixel value of the estimated low resolution image An update unit to The update unit uses the pixel value of each input low-resolution image, the amount of motion, and the pixel value of the estimated low-resolution image, and the input low-resolution image and the estimated low-resolution image; A low-resolution error image calculation unit that calculates a pixel value of a low-resolution error image that represents a difference in pixel values when the image is overlaid for each input low-resolution image, and pixel values of the plurality of low-resolution error images A high-resolution error image calculation unit that calculates a pixel value of a single high-resolution error image having a high resolution that represents an image obtained by combining the low-resolution error images, and the pixel value of the estimated output high-resolution image. An output high-resolution image updating unit that updates the pixel value of the estimated output high-resolution image by negative feedback of the pixel value of the high-resolution error image.

  According to this image processing apparatus, since the pixel value of the estimated output high resolution image is updated using the pixel value of one estimated low resolution image, the pixel values of a plurality of estimated low resolution images are used. As compared with the above, it is possible to reduce the processing load of generating a high-resolution image using a plurality of low-resolution images.

  In the image processing apparatus, the low-resolution image estimation unit may calculate a pixel value of the estimated low-resolution image by performing a smoothing process on the reference low-resolution image.

  According to this configuration, since the pixel value of the estimated low resolution image can be easily calculated, it is possible to further reduce the load of high resolution image generation processing using a plurality of low resolution images.

  In each of the image processing devices, the low resolution error image calculation unit moves each of the input low resolution images so as to overlap with the estimated low resolution image, thereby calculating a pixel value of each of the low resolution error images. The determination may be made so as to represent a difference in pixel value at each pixel position in the estimated low-resolution image.

  According to this configuration, each low resolution error image can be overlapped without being moved, so that it is possible to reduce the burden of calculating the pixel value of the high resolution error image.

  In the image processing apparatus, the high-resolution error image calculation unit calculates a total value of pixel values at the same pixel position of each low-resolution error image for each pixel position, and further represents the total value at each pixel position. It is preferable to calculate the pixel value of the high-resolution error image by executing sharpness enhancement processing and high-resolution processing on the image.

  According to this configuration, the amount of data to be subjected to sharpness enhancement processing and high resolution processing is reduced, so that it is possible to reduce the load of pixel value calculation processing of a high resolution error image. .

  In each of the image processing devices, regardless of the size of each pixel value of the high-resolution error image, the pixel value of the estimated output high-resolution image updated by the output high-resolution image update unit only once is the output It may be used as a pixel value of a high resolution image.

  According to this configuration, since the output high-resolution image is generated by one update, it is possible to further reduce the load of high-resolution image generation processing using a plurality of low-resolution images.

  In each of the image processing devices, a re-update low-resolution image that further updates the pixel value of the estimated low-resolution image using the pixel value of the estimated output high-resolution image updated by the output high-resolution image update unit. An estimation unit; and a control unit that controls each unit, wherein the control unit (i) causes the re-update low-resolution image estimation unit to calculate a pixel value of the updated estimated low-resolution image. A resolution image update process, and (ii) a re-update process in which the update unit updates the pixel value of the estimated output high-resolution image again using the pixel value of the updated estimated low-resolution image. It is good as well.

  According to this configuration, since the pixel value of the estimated output high resolution image is repeatedly updated, the estimated output high resolution image can be brought closer to an ideal image.

  In the image processing apparatus, the control unit calculates an evaluation value that reflects the magnitude of the difference between the estimated output high-resolution image and each input low-resolution image, and the evaluation value is a predetermined threshold value. The estimated low-resolution image update process (i) and the re-update process (ii) may be repeatedly executed until the value becomes less than or equal to the value.

  According to this configuration, the estimated output high resolution image can be made closer to an ideal image.

  In the image processing apparatus, the control unit calculates a magnitude of a difference between the estimated output high resolution image and each input low resolution image each time a pixel value of the entire estimated output high resolution image is calculated. The evaluation value to be reflected is calculated, and the estimated low-resolution image update process (i) and the re-update process (ii) are repeatedly executed until the evaluation value becomes larger than the previous evaluation value. Also good.

  According to this configuration, it is possible to prevent the re-update process from being repeated excessively.

  In the above-described image processing apparatus, the evaluation value may be a statistical value indicating a magnitude of deviation from zero of each pixel value of the high resolution error image.

  According to this configuration, the control unit can calculate an appropriate evaluation value.

  The present invention can be realized in various forms, for example, an image processing method and apparatus, a computer program for realizing the function of the method or apparatus, a recording medium on which the computer program is recorded, It can be realized in the form of a data signal including the computer program and embodied in a carrier wave.

Next, embodiments of the present invention will be described in the following order based on examples.
A. First embodiment:
B. Second embodiment:
C. Variations:

A. First embodiment:
A1. Device configuration:
FIG. 1 is a block diagram showing a configuration of an image processing system as an embodiment of the present invention. The image processing system 800 includes a printer 500 and a digital video camera 300. The printer 500 includes a data processing unit 100 and a print execution unit 700.

  The data processing unit 100 includes a CPU 110, a display unit 120, an operation unit 130, an interface unit (I / F unit) 150, and an internal storage device 200. These components are connected to each other via a bus 170.

  The display unit 120 has, for example, a liquid crystal display. The operation unit 130 includes an operation panel including a plurality of buttons, for example. The internal storage device 200 has a RAM, for example.

  The internal storage device 200 includes an image selection unit 215, a low resolution image estimation unit 220, a high resolution image estimation unit 230, a low resolution difference image generation unit 240, a movement amount calculation unit 250, and a high resolution difference image generation unit. 260, an output high-resolution image update unit 270, a print data generation unit 290, and a control unit 210 that controls these functional units. The functions of these components will be described later. Further, the function of each component is realized by a computer program executed by the CPU 110.

  The interface unit 150 has a plurality of input / output terminals and performs data communication with an external device such as the digital video camera 300. In addition, a print execution unit 700 is connected to the interface unit 150. The print execution unit 700 receives print data from the data processing unit 100 and prints an image. As the configuration of the print execution unit 700, various known configurations such as a configuration in which ink droplets are ejected for printing and a configuration in which printing is performed by a thermal transfer method can be employed.

  The digital video camera 300 generates a moving image. The generated moving image is transferred to the data processing unit 100. The data processing unit 100 generates a still image using a moving image. By the way, the moving image includes a plurality of still images (also called frame images). The data processing unit 100 generates a still image having a higher resolution than the frame image using a plurality of frame images. Each image includes a plurality of pixel values arranged in a matrix. However, a high resolution image has a higher pixel density than a low resolution image. Thus, in this specification, “high resolution” means that the pixel density is high, and “low resolution” means that the pixel density is low. Hereinafter, processing for generating a high-resolution still image using a plurality of still images is also referred to as “multi-image high-resolution processing”. The high resolution pixel density can be set to M times the low resolution pixel density (M is an arbitrary value larger than 1). However, M is preferably set to 2 or more, and in this embodiment, M = 2.

A2. Comparison example of high resolution processing for multiple images:
FIG. 2 is an explanatory diagram illustrating a comparative example of the multi-image high resolution processing. In the example of FIG. 2, one high-resolution output image Oimg is generated from three low-resolution frame images (hereinafter also referred to as “input image Iimg”). Hereinafter, in the figure, an image with a symbol LR (Low Resolution) represents a low resolution image, and an image with a symbol HR (High Resolution) represents a high resolution image.

  FIG. 3 is a flowchart showing the procedure of a comparative example of the multi-image high resolution processing. In step S100, the image selection unit 215 (FIG. 1) selects the input image Iimg according to a user instruction. As a method for selecting the input image Iimg, any method can be adopted. For example, the image selection unit 215 displays a moving image on the display unit 120, the user designates a favorite image (frame image) using the operation unit 130, and the image selection unit 215 performs the designated frame image (hereinafter “ A plurality of (in this case, three) frame images including “designated frame images” may be selected. At this time, a frame image whose order in time series is before and after the designated frame image may be selected, or another combination different from this may be selected.

  In the next step S110, the high resolution image estimation unit 230 (FIG. 1) generates a temporary output image Oimg using the input image Iimg. In the example of FIG. 2, a temporary output image Oimg is generated using the reference input image Rimg whose order along the time series is the center. Specifically, the high-resolution image estimation unit 230 generates a high-resolution temporary output image Oimg from the low-resolution reference input image Rimg using a known bilinear method.

  As will be described later, the final output image Oimg is generated by correcting the temporary output image Oimg. Therefore, the method for generating the temporary output image Oimg is not limited to the bilinear method, and various methods can be employed. For example, the nearest neighbor method may be used.

  Further, the reference input image Rimg may be determined according to other rules. For example, the first input image Iimg in the order along the time series may be used as the reference input image Rimg.

  In the next step S120, the movement amount calculation unit 250 (FIG. 1) calculates the relative movement amount of each input image Iimg. This relative movement amount is a value representing a relative position between the entire input image Iimg and the entire reference input image Rimg. As such a relative movement amount (also referred to as a relative motion amount), for example, a motion vector of the entire input image Iimg relative to the entire reference input image Rimg can be used. The relative movement amount is not limited to the movement amount representing the parallel movement of the image, and other movement amounts representing various movements can be employed. For example, you may employ | adopt the movement amount showing the motion containing a parallel movement and a rotational movement. In either case, various known methods can be employed as a method for calculating the relative movement amount. For example, a method of calculating a relative movement amount by extracting a feature point from each input image Iimg and calculating a movement amount of the feature point can be employed.

  In the next step S130, the low-resolution image estimation unit 220 (FIG. 1) uses the high-resolution temporary output image Oimg to generate three low-resolution estimated images Eimg (details will be described later). These estimated images Eimg represent input images that are consistent with the temporary output image Oimg. In other words, these estimated images Eimg represent images that each input image Iimg should have, assuming that the temporary output image Oimg is an optimal image. Here, each estimated image Eimg and each input image Iimg are associated one to one.

  Specifically, in step S130, the low resolution image estimation unit 220 generates each estimated image Eimg using the temporary output image Oimg and each relative movement amount. In the example of FIG. 2, the low-resolution image estimation unit 220 performs a blurring process on the temporary output image Oimg, further moves the image according to the relative movement amount, and executes the low-resolution process, thereby estimating the estimated image. Eimg is generated. Various known processes can be adopted as the blurring process (smoothing process). In the image moving process, the temporary output image Oimg is moved so as to overlap each input image Iimg. Various known processes can be adopted as the resolution reduction process. For example, an interpolation process such as a bilinear method or a nearest neighbor method can be employed. Further, when the pixel position of each estimated image Eimg overlaps the pixel position of the temporary output image Oimg, a process of simply thinning out the pixels may be employed.

  In the next step S140, the low resolution difference image generation unit 240 (FIG. 1) generates a low resolution difference image LDimg for each input image Iimg using the three input images Iimg and the three estimated images Eimg. To do. Each pixel value of the low resolution difference image LDimg is set to a difference (= Eimg−Iimg) of pixel values at the same pixel position between the input image Iimg and the estimated image Eimg. The meaning of this low resolution difference image LDimg can be explained as follows. When the relationship between the temporary output image Oimg and each input image Iimg is in an ideal state, that is, the temporary output image Oimg and each input image Iimg are the same in only the position (relative movement amount). In the case of representing an image, each pixel value of each low resolution difference image LDimg is a value close to zero. On the other hand, when the difference between the temporary output image Oimg and the ideal output image is large, that is, when the difference between the temporary output image Oimg and each input image Iimg is large, The pixel value of the low resolution difference image LDimg will depart from zero. Thus, the magnitude of the deviation of the pixel value of the low resolution difference image LDimg from zero means the magnitude of the difference between the temporary output image Oimg and the ideal output image. Therefore, if the pixel value of the low resolution difference image LDimg is fed back to the temporary output image Oimg, the temporary output image Oimg can be brought close to an ideal output image.

  In the next step S150, the high resolution difference image generation unit 260 (FIG. 1) generates a high resolution difference image HDimg for each low resolution difference image LDimg. Specifically, the high-resolution difference image generation unit 260 performs high-resolution processing on the low-resolution difference image LDimg, further performs sharpness enhancement processing, and then moves the image according to the estimated relative movement amount. Each high-resolution difference image HDimg is generated. As the resolution enhancement process, various known processes can be employed. For example, a bilinear method or a nearest neighbor method can be employed. As the sharpness enhancement process, various known processes can be employed. For example, processing using an unsharp mask may be employed. Furthermore, the image moving process is performed so that the high-resolution difference image HDimg overlaps the temporary output image Oimg. At this time, each pixel value of each high-resolution difference image HDimg is corrected so as to represent a pixel value at the same pixel position as that of the temporary output image Oimg. Various known interpolation processes can be employed as the process for correcting the pixel value.

  In the next step S160, the high resolution difference image generation unit 260 combines each high resolution difference image HDimg to generate one combined high resolution difference image CHDimg. Each pixel value of the combined high-resolution difference image CHDimg is set to the sum of pixel values at the same pixel position in each high-resolution difference image HDimg.

  In the next step S170, the output high-resolution image update unit 270 feeds back the synthesized high-resolution difference image CHDimg to the temporary output image Oimg. This feedback is performed by correcting each pixel value of the temporary output image Oimg according to the following equation (1).

  Pixel value after update = Pixel value before update + α × Difference pixel value ... (1)

  Here, the updated pixel value is the pixel value of the provisional output image Oimg after feedback. The pre-update pixel value is a pixel value of the temporary output image Oimg before feedback. The difference pixel value is a pixel value at the same pixel position in the combined high-resolution difference image CHDimg. α is a coefficient representing the strength of feedback. This coefficient α may be experimentally set in advance so that feedback does not become excessive. The sign α of the coefficient α is determined so as to realize negative feedback. In the present embodiment, the difference pixel value is obtained from a value obtained by subtracting the input image Iimg from the estimated image Eimg (Eimg−Iimg), and thus the coefficient α is a negative value.

  Through the above processing, the temporary output image Oimg can be updated to an image more suitable for the input image Iimg. After step S170, the process returns to step S130, and the processes of steps S130 to S170 are repeatedly executed. Then, when the pixel value of the combined high-resolution difference image CHDimg becomes sufficiently small, the control unit 210 determines that the process has converged and completes the update. The output image Oimg finally updated in this way is used as the final output image Oimg. In the example of FIG. 1, the print data generation unit 290 converts this final output image Oimg into print data. The control unit 210 outputs the print data to the print execution unit 700 via the interface unit 150. The print execution unit 700 prints an image according to the received print data.

  In this comparative example, since the blurring process (smoothing process) is executed in step S130, the absolute value of the pixel value of the sharp part of each input image Iimg in the low resolution difference image LDimg is the smooth part. The absolute value of the pixel value can be made larger. Furthermore, since the sharpness enhancement process is executed in step S150, it is possible to emphasize a sharp portion of each input image Iimg in the high resolution difference image HDimg. As a result, the updated temporary output image Oimg can be a sharp image that is consistent with each input image Iimg.

A3. First embodiment of the multi-image high resolution processing:
FIG. 4 is an explanatory diagram showing a first embodiment of the multi-image high resolution processing. There are four differences from the comparative example shown in FIG. The first difference is that the number of estimated images Eimga is one. The second difference is that each input image Iimg is moved so as to overlap with one estimated image Eimga when the low-resolution difference image LDimga is generated. The third difference is that when the synthesized high-resolution difference image CHDimga is generated, after the plurality of low-resolution difference images LDima are synthesized, the resolution-enhancing process is executed (in the comparative example, after the resolution is increased). Synthesis). The fourth difference is that the update is completed in one process.

  FIG. 5 is a flowchart showing the procedure of the multi-image high resolution processing. The processes in steps S200 to S220 are the same as the processes in steps S100 to S120 in FIG.

  In step S230, the low resolution image estimation unit 220 (FIG. 1) generates one estimated image Eimga using the input image Iimg. Specifically, the low-resolution image estimation unit 220 generates one estimated image Eimga by performing blurring processing (smoothing processing) on the reference input image Rimg. Here, the temporary output image Oimg and the estimated image Eimga are generated from the same reference input image Rimg (steps S210 and S230). Therefore, the low resolution image estimation unit 220 can generate an estimated image Eimga that is consistent with the temporary output image Oimg.

  In the next step S240, the low-resolution difference image generation unit 240 (FIG. 1) generates a low-resolution difference image LDimg for each input image Iimg using the three input images Iimg and one estimated image Eimga. To do. Specifically, the low-resolution difference image generation unit 240 moves the input image Iimg according to the relative movement amount, and generates a low-resolution difference image LDimg using the image after movement and the estimated image Eimga. In the image moving process, the input image Iimg is moved so as to overlap the estimated image Eimga. Further, each pixel value of the low resolution difference image LDimga is set to a difference (Eimga−Iimg) between each pixel value of the estimated image Eimga and a pixel value at the same pixel position in the input image Iimg after movement. When the pixel position of the input image Iimg after movement and the pixel position of the estimated image Eimga do not overlap, the pixel value in the image after movement is calculated by a known interpolation process. Thus, each of the three low-resolution difference images LDimg is composed of pixel difference values calculated at the pixel positions of one estimated image Eimga, so that the positions of the three low-resolution difference images LDimg are relative to each other. Are consistent with each other.

  In the next step S250, the high-resolution difference image generation unit 260 (FIG. 1) synthesizes the low-resolution difference images LDima to generate one synthesized low-resolution difference image CLDimg. Each pixel value of the combined low-resolution difference image CLDimg is set to the sum of the pixel values at the same pixel position in each low-resolution difference image LDimga. As described above, each low-resolution difference image LDimg is generated with reference to the pixel position of the common estimated image Eimga. Accordingly, the high-resolution difference image generation unit 260 can generate the combined low-resolution difference image CLDimg without moving each low-resolution difference image LDimga.

  In the next step S260, the high-resolution difference image generation unit 260 generates a combined high-resolution difference image CHDimga using the combined low-resolution difference image CLDimg. Specifically, the high-resolution difference image generation unit 260 performs a resolution enhancement process on the synthesized low-resolution difference image CLDimg, and further executes a sharpness enhancement process to thereby obtain a synthesized high-resolution difference image CHDimga. Is generated. The generated combined high-resolution difference image CHDimga can be superimposed on the temporary output image Oimg without being moved.

  In the next step S270, the output high-resolution image update unit 270 feeds back the combined high-resolution difference image CHDimga to the temporary output image Oimg. This process is the same as the process of step S170 in FIG.

  Thus, in the first embodiment, the temporary output image Oimg is corrected by the combined high-resolution difference image CHDimg representing the difference between the estimated image Eimga and each input image Iimg, so that it matches the input image Iimg. It is possible to generate a characteristic output image Oimg.

  In the first example, since the number of estimated images Eimga used for generating each low-resolution difference image LDimg is one, it is possible to reduce the load of the estimated image Eimga generation processing.

  In the first example, only the blurring process (smoothing process) is executed as the process for generating the estimated image Eimga. In the comparative example, the resolution reduction process is performed in addition to this. As a result, compared to the comparative example, it is possible to reduce the load of the generation process of the estimated image Eimga.

  Further, since the image obtained by performing the blurring process on the input image Iimg (reference input image Rimg) is used as the estimated image Eimga, the absolute value of the pixel value of the sharp portion of each input image Iimg in the low-resolution difference image LDimg Can be made larger than the absolute value of the pixel value of the smooth portion. As a result, by feeding back the difference, the updated output image Oimg can be a sharp image that is consistent with each input image Iimg.

  Furthermore, in the first embodiment, each of the input images Iimg is moved so as to overlap with one common estimated image Eimga, thereby generating each low-resolution difference image LDimg. Accordingly, the three low-resolution difference images LDima can be overlapped without being moved, so that it is possible to reduce the load of the generation process of the combined low-resolution difference image CLDima.

  In the first example, when the combined high-resolution difference image CHDimga is generated, the combining process (step S250) is executed before the resolution increasing process (FIG. 5: step S260). Therefore, since the image to be subjected to the high resolution processing can be made one, it is possible to reduce the load of the generation processing of the combined high resolution difference image CHDimga. Similarly, since the number of images to be subjected to sharpness enhancement processing can also be one, it is possible to further reduce the load of processing for generating the combined high-resolution difference image CHDimga.

B. Second embodiment:
B1. Device configuration:
FIG. 6 is a block diagram illustrating a configuration of an image processing system 800a according to the second embodiment. The only difference from the image processing system 800 of the first embodiment shown in FIG. 1 is that the internal storage device 200a has an update low-resolution image estimation unit 280. Other configurations are the same as those of the image processing system 800 shown in FIG. That is, in the printer 500a, the configuration of the data processing unit 100a other than the internal storage device 200a is the same as that of the printer 500 of the first embodiment.

  FIG. 7 is an explanatory diagram showing a second embodiment of the multi-image high resolution processing. FIG. 8 is a flowchart showing the procedure of the multi-image high resolution processing in the second embodiment. The difference from the first embodiment shown in FIGS. 4 and 5 is that the processes of steps S240 to S270 are repeatedly executed a plurality of times. The reason for repeating the series of processes (steps S240 to S270) is to make the temporary output image Oimg closer to an ideal output image.

  In the example of FIG. 8, first, the processes of steps S200 to S270 are executed as in the first embodiment shown in FIG. Next, in step S280, the control unit 210 determines whether or not to repeatedly perform the update. If it is determined that “Repeat”, in step S290, the update low-resolution image estimation unit 280 generates the estimated image Eimg using the temporary output image Oimg updated in step S270. Specifically, the update low-resolution image estimation unit 280 performs a blurring process on the updated temporary output image Oimg, and further performs a resolution reduction process to generate an estimated image Eimga ( FIG. 4). The new estimated image Eimga generated in this way represents an input image that is consistent with the updated temporary output image Oimg.

  Thereafter, based on the newly generated estimated image Eimga, the processes of steps S240 to S270 are executed. As a result, the temporary output image Oimg can be made closer to an ideal output image. Such a series of processes (steps S290, S240 to S270) are repeatedly executed until the control unit 210 determines that “stop the repetition” in step S280. When it is determined that “stop repetition”, the updated temporary output image Oimg is used as the final output image.

  In addition, various conditions can be employ | adopted as conditions for repetition in step S280. For example, the provisional output image Oimg may be updated (step S270) a predetermined number of times (for example, twice or three times).

  Further, the control unit 210 calculates an evaluation value that reflects the magnitude of the difference between the temporary output image Oimg and the ideal output image, that is, the magnitude of the difference between the temporary output image Oimg and each input image Iimg. And it is good also as judging whether it repeats based on the obtained evaluation value. As such an evaluation value, for example, a statistical value of the pixel value of the combined high-resolution difference image CHDimg can be employed. Specifically, it is possible to employ a statistical value (for example, the sum of squares, the maximum absolute value, or the sum of squares of pixel values in a predetermined central portion of the image) representing the magnitude of deviation from zero of each pixel value. it can. Further, the evaluation value may be calculated based on the synthesized low resolution difference image CLDimg instead of the synthesized high resolution difference image CHDimga. Here, the re-update process may be repeated until the evaluation value becomes equal to or less than a predetermined threshold value. In this way, the temporary output image Oimg can be made closer to the ideal output image. Further, the re-update process may be repeated until the evaluation value becomes larger than the evaluation value calculated from the previous provisional output image Oimg. In this way, it is possible to prevent the re-update process from being repeated excessively. In this case, the determination as to whether or not to repeat may be executed before step S270. In either case, the upper limit of the number of repetitions may be determined in advance.

C. Variations:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

Modification 1:
In each of the above embodiments, the estimated image Eimga is generated by executing the smoothing process (FIG. 5: step S230, FIG. 8: steps S230, S290), but the smoothing process may not be executed. For example, in step S230, the reference input image Rimg may be used as it is as the estimated image Eimga. In step S290, only the resolution reduction processing may be executed without executing the smoothing processing. However, if the smoothing process is executed, the updated temporary output image Oimg can be sharpened.

  Similarly, in each of the above embodiments, the synthesized high-resolution difference image CHDimga is generated by executing the sharpness enhancement process (FIG. 5, FIG. 8: Step S260), but the sharpness enhancement process may not be performed. Good. However, if the sharpness enhancement process is executed, it is possible to sharpen the updated temporary output image Oimg.

Modification 2:
In each of the above embodiments, the low-resolution difference image LDimg is generated by moving each input image Iimg (FIG. 5, FIG. 8: step S240), but the estimated image Eimga may be moved. However, if each input image Iimg is moved so as to overlap with one common estimated image Eimga, each low resolution difference image LDimg can be superimposed and synthesized without being moved. It is possible to reduce the load of the image LDimg synthesis process.

Modification 3:
In each of the above embodiments, the low resolution difference image Lldma is synthesized (FIG. 5, FIG. 8: Step S250) and then the high resolution processing (Step S260) is performed. The image LIdma may be synthesized after being subjected to high resolution processing. However, if the composition is performed before the resolution enhancement processing, the amount of data used for the resolution enhancement processing can be reduced, so the load of the generation processing of the synthesized high resolution difference image CHDimga is reduced. be able to.

  In each of the above embodiments, the sharpness enhancement process is executed after the resolution enhancement process in step S260 of FIGS. 5 and 8. Conversely, the resolution enhancement process is executed after the sharpness enhancement process. It is good as well. However, if the sharpness enhancement processing is executed after the high resolution processing, the sharpness enhancement processing can be executed on the high resolution image. Therefore, the synthesized high resolution difference image CHDimga, that is, the updated image is updated. The provisional output image Oimg can be made sharper.

Modification 4:
In each of the above-described embodiments, the coefficient α representing the strength of feedback is a predetermined fixed value, but it may be a variable value instead. For example, the output high-resolution image update unit 270 sets the coefficient α so that the absolute value of the coefficient α increases as the evaluation value described above (for example, the sum of squares of the pixel values of the combined high-resolution difference image CHDimga) increases. It is good to do.

Modification 5:
In each of the above-described embodiments, the combined high-resolution difference image CHDimga (combined low-resolution difference image CLDimg) is generated with the same weight for each low-resolution difference image Lldma (that is, each input image Iimg). Specifically, the total value (total value with equal weight) of the pixel values of each low-resolution difference image LDimga is used as the pixel value of the combined low-resolution difference image CLDima. Instead, a different weight may be assigned to each low resolution difference image Lldma (each input image Iimg). In this way, when the images are different between the input images Iimg, it is possible to suppress the difference from being emphasized. For example, if the weight of the reference input image Rimg is larger than the weight of the other input images Iimg, the output image Oimg can be an image that appropriately represents the reference input image Rimg.

  In general, the combined high-resolution difference image CHDimga may be an image obtained by combining the low-resolution difference images LDima. Here, the “image obtained by combining a plurality of images” means that each pixel value of the image obtained by combining is a function of the pixel values of the plurality of base images. Similarly, as the pixel value of the combined low-resolution difference image CLDimg, the total value of the pixel values at the same pixel position of each low-resolution difference image LDima can be used. Here, the “total value of a plurality of pixel values” is a function of a plurality of pixel values, and is a value having a positive correlation with the sum of pixel values. As the total value, if a total value with equal weights is used as in each of the above-described embodiments, it is possible to reduce the load of the process of generating the combined low-resolution difference image CLDimga.

Modification 6:
In each of the above embodiments, three are used as the number of input images Iimg used for the multi-image high resolution processing, but any number other than three can be employed. However, if an excessively large number is used, the processing load becomes excessively high. Therefore, the number of input images Iimg is preferably between 2 and 5.

Modification 7:
In each of the embodiments described above, the data processing units 100 and 100a are provided in the printer. Instead, the data processing units 100 and 100a may be configured using a general personal computer. The data processing units 100 and 100a may be provided in various other devices such as portable information terminals, mobile phones, digital video cameras, and digital still cameras. As described above, in each of the above-described embodiments, the load of the multi-image high resolution processing is reduced. Therefore, not only a personal computer having a relatively high calculation processing capability, but also various devices that cannot be said to have a high calculation processing capability, it is possible to realize a multi-image high resolution processing. The generated high-resolution output image Oimg can be used for various purposes other than printing. For example, it may be used for display on a monitor, and an output image file storing an output image Oimg (more precisely, output image data representing the output image Oimg) is stored in an external storage device (for example, a hard disk drive). ). The output image data may be transmitted to other devices connected to the data processing units 100 and 100a.

Modification 8:
In each of the above embodiments, a high-resolution still image is generated from a moving image. Instead, a high-resolution still image may be generated from a plurality of low-resolution still images.

Modification 9:
In each of the above embodiments, a part of the configuration realized by software may be replaced by hardware, and conversely, a part of the configuration realized by hardware may be replaced by software. .

1 is a block diagram illustrating a configuration of an image processing system as an embodiment of the present invention. It is explanatory drawing which shows the comparative example of the multiple image high resolution process. It is a flowchart which shows the procedure of the comparative example of the double image high resolution process. It is explanatory drawing which shows 1st Example of the multi-image high resolution process. It is a flowchart which shows the procedure of the resolution enhancement process of a multiple image. It is a block diagram which shows the structure of the image processing system 800a in 2nd Example. It is explanatory drawing which shows 2nd Example of the multiple-image high resolution process. It is a flowchart which shows the procedure of the multi-image high resolution process in 2nd Example.

Explanation of symbols

100, 100a ... data processing unit 110 ... CPU
DESCRIPTION OF SYMBOLS 120 ... Display part 130 ... Operation part 150 ... Interface part 170 ... Bus 200, 200a ... Internal storage device 210 ... Control part 215 ... Image selection part 220 ... Low Resolution image estimation unit 230 ... High resolution image estimation unit 240 ... Low resolution difference image generation unit 250 ... Movement amount calculation unit 260 ... High resolution difference image generation unit 270 ... Output high resolution image update Unit 280 ... low-resolution image estimation unit for update 290 ... print data generation unit 300 ... digital video camera 500, 500a ... printer 700 ... print execution unit 800, 800a ... image processing system

Claims (11)

An image processing apparatus for generating a high-resolution output high-resolution image having a relatively high pixel density from a plurality of low-resolution input low-resolution images having a relatively low pixel density,
A high-resolution image estimation unit that calculates a pixel value of an estimated output high-resolution image that is an estimated image of the output high-resolution image using a reference low-resolution image that is one image among the plurality of input low-resolution images When,
A low-resolution image estimation unit that creates one low-resolution estimated low-resolution image using the reference low-resolution image;
A motion amount calculating unit that calculates a relative motion amount of the plurality of input low resolution images;
An update unit that updates pixel values of the estimated output high-resolution image using pixel values of the plurality of input low-resolution images and pixel values of the estimated low-resolution image;
With
The update unit
A pixel when the input low resolution image and the estimated low resolution image are superimposed using the pixel value of the input low resolution image, the amount of motion, and the pixel value of the estimated low resolution image. A low-resolution error image calculation unit that calculates a pixel value of a low-resolution error image representing a difference in values for each input low-resolution image;
A high-resolution error image calculation unit that calculates pixel values of a single high-resolution error image representing a high-resolution image that represents an image obtained by combining the low-resolution error images using pixel values of the plurality of low-resolution error images;
An output high-resolution image update unit that updates the pixel value of the estimated output high-resolution image by negative feedback of the pixel value of the high-resolution error image with respect to the pixel value of the estimated output high-resolution image;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The low-resolution image estimation unit calculates a pixel value of the estimated low-resolution image by performing a smoothing process on the reference low-resolution image. The image processing apparatus according to claim 1 or 2,
The low-resolution error image calculation unit moves each of the input low-resolution images so as to overlap the estimated low-resolution image, thereby obtaining a pixel value of each low-resolution error image in the estimated low-resolution image. Determine to represent the difference in pixel values at each pixel position;
Image processing device. The image processing apparatus according to claim 3,
The high-resolution error image calculation unit calculates a total value of pixel values at the same pixel position of each low-resolution error image for each pixel position, and further sharpness enhancement is performed on an image represented by the total value at each pixel position. An image processing apparatus that calculates a pixel value of the high-resolution error image by executing processing and high-resolution processing. An image processing apparatus according to any one of claims 1 to 4,
Regardless of the size of each pixel value of the high-resolution error image, the pixel value of the estimated output high-resolution image updated only once by the output high-resolution image update unit is used as the pixel value of the output high-resolution image. Used,
Image processing device. The image processing apparatus according to any one of claims 1 to 4, further comprising:
A re-updating low-resolution image estimation unit that updates the pixel value of the estimated low-resolution image using the pixel value of the estimated output high-resolution image updated by the output high-resolution image update unit;
A control unit for controlling each unit,
The controller is
(i) an estimated low-resolution image update process that causes the re-update low-resolution image estimation unit to calculate pixel values of the updated estimated low-resolution image;
(ii) re-update processing in which the update unit updates the pixel value of the estimated output high-resolution image again using the pixel value of the updated estimated low-resolution image;
An image processing apparatus that executes The image processing apparatus according to claim 6,
The control unit calculates an evaluation value that reflects the magnitude of the difference between the estimated output high resolution image and each input low resolution image, and until the evaluation value is equal to or less than a predetermined threshold value, An image processing apparatus that repeatedly executes the estimated low-resolution image update process (i) and the re-update process (ii). The image processing apparatus according to claim 6,
The control unit calculates an evaluation value that reflects the magnitude of the difference between the estimated output high-resolution image and each input low-resolution image every time the pixel value of the entire estimated output high-resolution image is calculated. In addition, an image processing apparatus that repeatedly executes the estimated low-resolution image update process (i) and the re-update process (ii) until the evaluation value becomes larger than the previous evaluation value. The image processing apparatus according to claim 7 or 8, wherein
The evaluation value is a statistic value indicating the amount of deviation from zero of each pixel value of the high resolution error image.
Image processing device. An image processing method for generating a high-resolution output high-resolution image having a relatively high pixel density from a plurality of low-resolution input low-resolution images having a relatively low pixel density,
(A) calculating a pixel value of an estimated output high-resolution image that is an estimated image of the output high-resolution image using a reference low-resolution image that is one image among the plurality of input low-resolution images; ,
(B) calculating a pixel value of one low-resolution estimated low-resolution image using the reference low-resolution image;
(C) calculating a relative motion amount of the plurality of input low resolution images;
(D) updating pixel values of the estimated output high resolution image using pixel values of the plurality of input low resolution images and pixel values of the estimated low resolution image;
With
The step (d)
(D1) The input low resolution image and the estimated low resolution image are superimposed using the pixel value of each input low resolution image, the motion amount, and the pixel value of the estimated low resolution image. Calculating a pixel value of a low-resolution error image representing a difference in pixel value for each input low-resolution image;
(D2) calculating pixel values of one high-resolution error image having a high resolution representing an image obtained by combining the low-resolution error images using pixel values of the plurality of low-resolution error images;
(D3) updating the pixel value of the estimated output high resolution image by negative feedback of the pixel value of the high resolution error image with respect to the pixel value of the estimated output high resolution image;
Including an image processing method. A computer program for causing a computer to execute image processing for generating a high-resolution output high-resolution image having a relatively high pixel density from a plurality of low-resolution input low-resolution images having a relatively low pixel density,
(A) Using a reference low-resolution image that is one of the plurality of input low-resolution images, calculate a pixel value of an estimated output high-resolution image that is an estimated image of the output high-resolution image. Function
(B) a function of causing a computer to calculate a pixel value of one estimated low resolution image having a low resolution using the reference low resolution image;
(C) a function of causing a computer to calculate a relative motion amount of the plurality of input low-resolution images;
(D) A function of causing a computer to update the pixel value of the estimated output high resolution image using the pixel value of the plurality of input low resolution images and the pixel value of the estimated low resolution image;
Is a computer program that realizes
The function (d) is
(D1) Using the pixel value of each input low-resolution image, the amount of motion, and the pixel value of the estimated low-resolution image to the computer, the input low-resolution image and the estimated low-resolution image A function of calculating a pixel value of a low-resolution error image representing a difference in pixel values when superimposed, for each input low-resolution image;
(D2) a function of causing a computer to calculate pixel values of a single high-resolution error image having a high resolution representing an image obtained by combining the low-resolution error images using pixel values of the plurality of low-resolution error images; ,
(D3) A function of causing the computer to update the pixel value of the estimated output high-resolution image by negative feedback of the pixel value of the high-resolution error image with respect to the pixel value of the estimated output high-resolution image;
Including computer programs.

Images