JP2006221221A - Generation of high resolution image using two or more low resolution image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for acquiring a high definition image while reducing the increase of a noise accompanied with improved resolution due to the composition of low resolution images. <P>SOLUTION: An image processor includes an edge detecting part and an improved resolution processing part. The improved resolution processing part selects and executes improved resolution in one improved resolution processing mode from among a plurality of improved resolution processing modes in accordance with the scale of the edge quantity of a processing position as the processing object of improved resolution processing. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for generating a high resolution image from a plurality of low resolution images.

  A moving image photographed by a digital video camera includes a plurality of frame images each representing one scene of the moving image. Conventionally, a process for generating a still image having a higher pixel density (that is, higher resolution) than a frame image using a plurality of frame images is known (see, for example, Patent Document 1). This processing is also called “high resolution processing” or “high definition processing”.

  In the present specification, the phrase “high definition” means that not only the resolution (pixel density) is high but also the amount of image information is large. Therefore, for example, when the resolution is doubled by the simple padding process, the resolution is doubled, but the definition is not changed.

Japanese Patent Laid-Open No. 2004-272751

  In this high resolution processing, a single high resolution image is created by combining a plurality of frame images, so that a higher definition image can be obtained. However, in an image region (so-called low-frequency image region) in which the image component is smooth in the frame image, the definition of the image is not greatly improved by increasing the resolution. On the other hand, in a smooth image area, noise is enhanced due to high resolution, and image quality may deteriorate. In particular, since a moving image includes a considerable amount of noise, there is a tendency that a problem that noise in a smooth image region increases by combining a plurality of frame images.

  Such a problem is not limited to a case where a still image is generated using a moving image, but is generally a problem common to a case where a high resolution image is generated using a plurality of low resolution images.

  The present invention has been made to solve the above-described problems, and provides a technique capable of obtaining a high-definition image while reducing an increase in noise caused by a higher resolution by combining low-resolution images. The purpose is to do.

An image processing apparatus according to the present invention includes:
An image processing apparatus that creates a single high-resolution image using a plurality of low-resolution images including a reference image,
An edge detection unit that detects edges in the reference image by analyzing the reference image;
A high-resolution processing unit that executes a high-resolution processing for creating one high-resolution image based on the plurality of low-resolution images;
With
The high resolution processing has a plurality of high resolution processing modes having different high definition effects as high resolution processing modes each using at least one of the plurality of low resolution images.
The high-resolution processing unit uses one high-resolution processing mode from among the plurality of high-resolution processing modes according to the size of the edge amount of the processing position that is the processing target of the high-resolution processing. Select and execute high resolution.

  In a processing mode having a large high-definition effect, there is a tendency for noise to be emphasized although the definition becomes high. Conversely, in a processing mode with a small high-definition effect, the definition is not so high, but the noise is not so emphasized. By the way, since the change of the pixel value is large at the position where the edge is large, a high-definition image can be obtained by selectively using a processing mode which has a large effect of high-definition. On the other hand, since the change in the pixel value is small at the position where the edge is small, there is no great difference in the resolution actually obtained by the different high resolution processing modes. In addition, if the resolution is increased in a processing mode having a high definition effect when noise is included at a position where the edge is small, the noise may be emphasized and the image quality may be deteriorated. In the present invention, a plurality of high resolution processing modes having different high definition effects are prepared, and one of them is selectively executed according to the size of the edge amount at the processing position. A high-definition image can be obtained while reducing an increase in noise accompanying the increase in resolution of the image.

The plurality of high resolution processing modes are:
A combined high-resolution processing mode for creating the high-resolution image by combining the plurality of low-resolution images;
A sharpness-enhancing and high-resolution processing mode for executing a simple high-resolution process and a sharpness-enhancing process for increasing the resolution of the reference image using only the reference image;
A noise removal and high resolution processing mode for performing simple high resolution processing and noise removal processing for increasing the resolution of the reference image using only the reference image;
Have
The high resolution processing unit
(I) Select and execute high resolution in the composite high resolution processing mode at a position where the edge amount is large,
(Ii) In the position where the edge amount is medium, the high resolution is selected and executed by the sharpness enhancement high resolution processing mode;
(Iii) In a position where the edge amount is small, high resolution in the noise removal high resolution processing mode may be selected and executed.

  According to this configuration, a high-definition image can be obtained at a position where the edge amount is large, a low-noise image can be obtained at a position where the edge amount is small, and the definition and noise can be obtained at a position where the edge amount is medium. An intermediate image can be obtained in both quantities. Therefore, it is possible to obtain a high-resolution image with a good balance between definition and noise.

  In the sharpness high-resolution processing mode, the strength of the sharpness enhancement processing may be set larger as the edge amount is larger.

  The higher the strength of the sharpness enhancement processing, the higher the definition, and the noise tends to be less emphasized. Therefore, according to this configuration, it is possible to obtain a high-resolution image with a better balance between definition and noise.

  The present invention can be realized in various forms, for example, an image processing method and apparatus, a computer program for realizing the function of the method or apparatus, a recording medium on which the computer program is recorded, It can be realized in the form of a data signal including the computer program and embodied in a carrier wave.

Next, embodiments of the present invention will be described in the following order based on examples.
A. Example:
B. Details of composite high-resolution processing:
C. Variations:

A. Example:
FIG. 1 is a block diagram showing a configuration of an image processing system as an embodiment of the present invention. The image processing system 800 includes a printer 500 and a digital video camera 300. The printer 500 includes a data processing unit 100 and a print execution unit 700 (printing mechanism). Although not shown, the printer 500 includes a display unit for displaying images and operation guidance, and an operation unit for the user to specify the operation of the printer 500. The operation of selecting a plurality of frames used for increasing the resolution from the moving image is performed using these display units and operation units.

  The data processing unit 100 receives a moving image from the video camera 300, performs a resolution increasing process (described later), generates a high resolution image, and generates print data thereof. The print execution unit 700 receives this print data and performs printing. As the printing execution unit 700, various known printing mechanisms such as an ink jet printing mechanism and a thermal transfer printing mechanism can be adopted.

  FIG. 2 is a block diagram illustrating an internal configuration of the data processing unit 100. The data processing unit 100 includes an edge detection unit 420, a plurality of line buffers 430, a resolution enhancement processing unit 440, a print data generation unit 450, and a motion vector detection unit 460. The functions of the edge detection unit 420, the resolution enhancement processing unit 440, the print data generation unit 450, and the motion vector detection unit 460 are realized by a CPU (not shown) executing a computer program. Therefore, these units can be called “program modules” or simply “modules”.

  In the present embodiment, one high-resolution image Oimg is generated using three frame images Iimg1 to 3 (hereinafter referred to as “input images”), and printing is executed. The three input images Iimg1 to 3 are selected from the moving images. Here, three frame images separated by a predetermined time interval Δt are selected. However, the number of input images and the time interval can be arbitrarily set.

  The central image Iimg2 of the three input images Iimg1 to 3 is used as a reference input image Rimg (also referred to as “reference image”). As will be described later, the reference input image Rimg is used for various purposes. That is, (i) it becomes an object of edge detection, and (ii) it is used as an original image for simple high resolution processing. As the reference input image Rimg, it is possible to select any image from among the plurality of input images Iimg.

  The line buffer 430 is a buffer that holds image data of three lines Li-1, Li, Li + 1 (i is a line number) of the reference input image Rimg for edge detection processing. The edge detection unit 420 detects the presence or absence of an edge from the image data for these three lines (details will be described later).

  The high resolution processing unit 440 includes a noise removal high resolution processing unit 441, a sharpness enhancement high resolution processing unit 443, and a combined high resolution processing unit 444. The noise removal and high resolution processing unit 441 performs simple high resolution and noise removal processing using only the reference input image Rimg. On the other hand, the sharpness enhancement and resolution enhancement processing unit 443 performs simple resolution enhancement and sharpness enhancement processing using only the reference input image Rimg. The combined high resolution processing unit 444 performs high resolution by combining the three input images Iimg1 to Iimg1 to 3. In this specification, the processing performed by the noise removal and higher resolution processing unit 441 is referred to as “noise removal and higher resolution mode”, and the processing performed by the sharpness enhancement and higher resolution processing unit 443 is referred to as “sharpness processing and higher resolution mode”. The processing performed by the resolution enhancement processing unit 444 is referred to as “composite resolution enhancement mode”. The high resolution processing unit 440 selects and executes one of a noise removal high resolution mode, a sharpness-enhanced high resolution mode, and a combined high resolution mode according to the edge detection result. In the present embodiment, the resolution is increased for each line Li to be processed.

  The high-resolution image Oimg created by the high resolution is supplied to the print data generation unit 450. The print data generation unit 450 generates print data of the high resolution image Oimg and supplies it to the print execution unit 700 (FIG. 1) to execute printing.

  FIG. 3 is an explanatory diagram illustrating an example of a relationship between pixel positions of a low resolution image and a high resolution image. In FIG. 3, the pixel position of the low-resolution image is indicated by a circle, and the pixel position of the high-resolution image is indicated by a cross. In this example, the pixel pitch Phigh of the high resolution image is half of the pixel pitch Plow of the low resolution image. That is, the high resolution image has twice the resolution (that is, twice the pixel density) of the low resolution image. The pixel position of the high resolution image is shifted from the pixel position of the low resolution image in the horizontal direction and the vertical direction by 1/2 of the high resolution pixel pitch Phigh.

  In FIG. 3, a frame indicated by a broken line indicates a range of four high-resolution pixels associated with each low-resolution pixel. In the simple high resolution, the pixel value of one low resolution image is copied to the four high resolution pixels in the broken line. That is, the pixel value of the high resolution pixel in the broken line is set to be the same as the pixel value of the low resolution pixel. Therefore, with this simple high resolution, the resolution is increased, but the definition is not changed. However, in the simple high resolution processing, the pixel value of the high resolution pixel may be calculated using an interpolation calculation by a bilinear method or the like. When interpolation is used for higher resolution, the definition is slightly improved. In the noise removal high resolution processing, noise removal processing is executed along with simple high resolution. When the noise removal process is performed, the definition decreases. Further, in the sharpness enhancement high resolution processing, sharpness enhancement processing is executed along with simple high resolution. When sharpness enhancement processing is performed, the definition is improved. On the other hand, in the composite high-resolution processing, pixel values of high-resolution pixels are calculated using not only the reference input image Rimg but also a plurality of input images Iimg1 to 3 (details will be described later). Therefore, in the composite high resolution processing, not only the resolution but also the definition is improved. The effect of high definition is the highest in the combined high resolution processing, the smallest in the noise removal high resolution processing, and the sharpness-enhanced high resolution is between these.

  FIG. 4 is an explanatory diagram illustrating another example of the relationship between the pixel positions of the low resolution image and the high resolution image. In this example, the position of the quarter pixel in the high resolution image matches the position of the pixel in the low resolution image. As the relationship between the pixel positions of the low-resolution image and the high-resolution image, various things other than those shown in FIGS. 3 and 4 can be adopted. However, in the embodiment described below, the positional relationship shown in FIG. 3 is used.

  In general, the pixel pitch Phigh of the high resolution image can be set to 1 / M times the pixel pitch Plow of the low resolution image (M is an arbitrary value larger than 1). However, M is preferably set to 2 or more, and in this embodiment, M = 2.

  FIG. 5 is a flowchart showing the overall procedure of the resolution enhancement processing in this embodiment. In step S40, various initializations are performed. At this time, the target line number Nline indicating the target line of the resolution enhancement process and the target pixel number Npix indicating the target pixel are also initialized to zero. The target line Nline and the target pixel Npix indicate lines and pixels (low resolution pixels) in the reference image Rimg.

  In step S41, the motion vector detection unit 460 calculates the motion vector (relative movement amount) of each input image Iimg. This motion vector is a vector representing the relative position of the entire reference input image Rimg and the other input images (Iimg1 or Iimg3). Therefore, when there are three input images, two motion vectors are obtained. These motion vectors are used in the synthetic high resolution processing. In general, when the number of input images Iimg is N + 1 (N is an integer equal to or greater than 1) including the reference input image Rimg, N motion vectors are calculated.

  In step S42, image data for three lines centered on the target line Nline is selected from the reference input image Rimg and input to the line buffer 430 (FIG. 2). If Nline = 0 indicates the upper end line of the image area, when Nline = 0, the line Li-1 before the target line Nline (= Li) does not actually exist. In this case, the image data of the target line Nline (= Li) may be copied and used as the image data of the line Li-1. Or you may make it start the process after step S42 from Nline = 1. The same applies to the target pixel Npix.

  In step S43, the edge detection unit 420 detects the edge of the target pixel Npix in the reference image Rimg using the image data in the line buffer 430. FIG. 6 is an explanatory diagram showing a method of edge detection. FIG. 6A shows primary differential filters FLx and FLy for edge detection in the x and y directions. FIG. 6B shows Sobel filters FLx and FLy for edge detection in the x and y directions. In the present embodiment, primary differential filters FLx and FLy are used. However, as the edge detection filter (also referred to as “edge amount calculation filter”), any filter other than these can be used, and for example, a secondary differential filter can also be used.

  By performing filter processing on the target pixel Npix using the two first-order differential filters FLx and FLy, the edge amount component Edx in the x direction and the edge amount component Edy in the y direction of the target pixel Npix can be calculated. The edge amount Ed and the edge direction θ of the target pixel Npix can be calculated, for example, according to the equation shown in FIG. This edge amount Ed is an edge amount (also referred to as “total edge amount”) obtained by combining the edge amount component Edy in the x direction and the edge amount component Edy in the y direction. As in this example, as the edge amount used for edge detection (edge amount determination), it is preferable to use a total edge amount including edge amount components in two directions. For example, the edge amount obtained using a Laplacian filter can be considered as a kind of total edge amount.

  The determination of the edge amount is performed by comparing the edge amount Ed with predetermined thresholds Eth1 and Eth2. That is, when the edge amount Ed is equal to or smaller than the first threshold value Eth1, the edge amount is determined to be small, and when the edge amount Ed is between the two threshold values Eth1, Eth2, the edge amount is determined to be medium, and the second threshold value is determined. When it is greater than Eth2, it is determined that the edge amount is large.

  If it is determined in step S44 of FIG. 5 that the edge of the target pixel is small, the noise removal and high resolution processing unit 441 executes the high resolution processing in step S45a and is associated with the target pixel. Pixel values of a plurality of high resolution pixels (four high resolution pixels surrounded by a broken line in FIG. 3) are calculated. Specifically, first, simple high resolution processing is performed on the reference input image Rimg to double the resolution, and then noise removal processing is executed. The simple high resolution processing is realized, for example, by copying pixel values of a low resolution image to four high resolution pixels. Instead of copying the pixel value, the pixel value of the high resolution pixel may be determined by interpolating the pixel values of a plurality of low resolution pixels adjacent to the high resolution pixel. That is, in the simple high resolution process, only the reference input image Rimg is used, and the pixel value of each high resolution pixel can be generated by referring to one or more nearest pixels. As the noise removal process, for example, a smoothing process can be used. Note that the order of the simple resolution enhancement process and the noise removal process may be reversed.

  A smooth image portion having a small edge amount tends to include a lot of noise. Therefore, if noise removal processing is performed at a position where the edge amount is small, a smooth high-resolution image with little noise can be obtained.

  If it is determined in step S44 that the edge amount of the target pixel is medium, the sharpness enhancement high resolution processing unit 443 executes the high resolution processing in step S45b. Specifically, first, the reference input image Rimg is subjected to the simple high resolution processing to double the resolution, and then the sharpness enhancement processing is executed. As the sharpness enhancement processing, for example, processing using an unsharp mask can be used. Note that the order of the simple resolution enhancement processing and the sharpness enhancement processing may be reversed.

Sharpness enhancement processing using an unsharp mask is expressed by the following equation.
S ′ = S + k (S−U)
Here, S ′ is a pixel value after sharpness enhancement processing, S is an original pixel value, U is a pixel value after unsharpening processing (smoothing processing), and k is a coefficient.

  The strength of the sharpness enhancement process can be adjusted, for example, by changing the coefficient k in the above equation. The strength of the sharpness enhancement process may be adjusted according to the size of the edge amount. For example, a medium edge amount range (Eth1 to Eth2 range) is further divided into several levels (eg, 3 to 5 levels), and a coefficient k suitable for the edge amount level is set in advance. Can be kept. In this case, it is preferable to increase the strength of sharpness enhancement processing (that is, the value of the coefficient k) as the edge amount increases. By doing so, the definition can be made higher at a position where the edge amount is larger, and noise can be hardly emphasized at a position where the edge amount is smaller.

  If it is determined in step S44 that the edge amount of the target pixel is large, in step S45c, the combined high resolution processing unit 444 executes the high resolution processing, and a plurality of high resolution pixels associated with the target pixel. A pixel value of (four high resolution pixels surrounded by a broken line in FIG. 3) is calculated. In this composite high resolution processing, not only the reference input image Rimg but also the plurality of input images Iimg1 to 3 are used together to calculate pixel values of high resolution pixels. Details of the composite high resolution processing will be described later.

  The pixel values of the resolution-enhanced image created in this way are sequentially supplied from the resolution-enhancement processing unit 440 to the print data generation unit 450 (FIG. 2).

  In step S47 of FIG. 5, the target pixel number Npix is updated. When the target pixel reaches the end of each line, the target line number Nline is also updated. If the processing for all the pixels has not been completed, the process returns from step S48 to step S42, the image data in the line buffer 430 is updated as necessary, and the processes of steps S43 to S48 are repeated.

  As described above, in the present embodiment, since the synthetic high resolution processing is executed at the pixel position where the edge amount is determined to be large, a higher definition image can be obtained for the image portion where the pixel value changes sharply. it can. On the other hand, at the pixel position where the edge amount is determined to be small, the noise removal and high resolution processing is executed, so that noise can be reduced for the image portion where the change in pixel value is slow. Furthermore, since the sharpness-enhancing and high-resolution processing is executed at the pixel position where the edge amount is determined to be medium, an image with a good balance between sharpness and noise can be obtained. In addition, the noise removal high resolution processing and sharpness enhancement high resolution processing have an advantage that the overall processing time can be shortened because the processing time is shorter than the synthetic high resolution processing.

  In the above embodiment, the high resolution processing mode is selected for each individual low resolution pixel, but instead, the high resolution processing mode may be selected for each line. For example, for each line of the reference input image Rimg, an edge amount index value (for example, a sum of edge amounts or a square sum of edge amounts) having a correlation with the sum of edge amounts is calculated, and the edge amount index value is calculated. If the size of the edge amount is determined accordingly, it is possible to select the high resolution processing mode for each line.

B. Details of composite high-resolution processing:
FIG. 7 is a block diagram showing an internal configuration of the composite high resolution processing unit 444. The combined high resolution processing unit 444 includes a control unit 210, an image selection unit 215, a low resolution image estimation unit 220, a high resolution image estimation unit 230, a low resolution difference image generation unit 240, and a movement amount calculation unit 250. A high-resolution difference image generation unit 260, and an output high-resolution image update unit 270. The functions of these components will be described later.

  FIG. 8 is an explanatory diagram showing the contents of the composite high resolution processing. In the example of FIG. 8, one high-resolution output image Oimg is generated from three input images Iimg. Hereinafter, in the figure, an image with a symbol LR (Low Resolution) represents a low resolution image, and an image with a symbol HR (High Resolution) represents a high resolution image.

  FIG. 9 is a flowchart showing the procedure of the composite high resolution processing. In step S110, the high resolution image estimation unit 230 (FIG. 7) generates a temporary output image Oimg using the input image Iimg. In the example of FIG. 8, the temporary output image Oimg is generated using the reference input image Rimg having the center in the order along the time series. Specifically, the high-resolution image estimation unit 230 generates a high-resolution temporary output image Oimg from the low-resolution reference input image Rimg using a known bilinear method.

  As will be described later, the final output image Oimg is generated by correcting the temporary output image Oimg. Therefore, the method for generating the temporary output image Oimg is not limited to the bilinear method, and various methods can be employed. For example, the nearest neighbor method may be used.

  Further, the reference input image Rimg may be determined according to other rules. For example, the first input image Iimg in the order along the time series may be used as the reference input image Rimg.

  In the next step S120, the movement amount calculation unit 250 (FIG. 7) calculates the relative movement amount of each input image Iimg. This relative movement amount is a value representing a relative position between the entire input image Iimg and the entire reference input image Rimg. As such a relative movement amount (also referred to as a relative motion amount), for example, a motion vector of the entire input image Iimg relative to the entire reference input image Rimg can be used. The relative movement amount is not limited to the movement amount representing the parallel movement of the image, and other movement amounts representing various movements can be employed. For example, you may employ | adopt the movement amount showing the motion containing a parallel movement and a rotational movement. In either case, various known methods can be employed as a method for calculating the relative movement amount. For example, a method of calculating a relative movement amount by extracting a feature point from each input image Iimg and calculating a movement amount of the feature point can be employed.

  In the processing procedure shown in FIG. 5, a motion vector (relative movement amount) is calculated in step S11. Therefore, in step S120, there is no need to perform the calculation again, and the value obtained in step S11 can be used as it is. In this case, step S120 and the movement amount calculation unit 250 can be omitted.

  In the next step S130, the low resolution image estimation unit 220 (FIG. 7) generates three low resolution estimated images Eimg using the high resolution temporary output image Oimg (details will be described later). These estimated images Eimg represent input images that are consistent with the temporary output image Oimg. In other words, these estimated images Eimg represent images that each input image Iimg should have, assuming that the temporary output image Oimg is an optimal image. Here, each estimated image Eimg and each input image Iimg are associated one to one.

  Specifically, in step S130, the low resolution image estimation unit 220 generates each estimated image Eimg using the temporary output image Oimg and each relative movement amount. In the example of FIG. 8, the low-resolution image estimation unit 220 performs a blurring process on the temporary output image Oimg, further moves the image according to the relative movement amount, and executes the low-resolution process, thereby estimating the estimated image. Eimg is generated. Various known processes can be adopted as the blurring process (smoothing process). In the image moving process, the temporary output image Oimg is moved so as to overlap each input image Iimg. Various known processes can be adopted as the resolution reduction process. For example, an interpolation process such as a bilinear method or a nearest neighbor method can be employed. Further, when the pixel position of each estimated image Eimg overlaps the pixel position of the temporary output image Oimg, a process of simply thinning out the pixels may be employed.

  In the next step S140, the low resolution difference image generation unit 240 (FIG. 7) generates a low resolution difference image LDimg for each input image Iimg using the three input images Iimg and the three estimated images Eimg. To do. Each pixel value of the low resolution difference image LDimg is set to a difference (= Eimg−Iimg) of pixel values at the same pixel position between the input image Iimg and the estimated image Eimg. The meaning of this low resolution difference image LDimg can be explained as follows. When the relationship between the temporary output image Oimg and each input image Iimg is in an ideal state, that is, the temporary output image Oimg and each input image Iimg are the same in only the position (relative movement amount). In the case of representing an image, each pixel value of each low resolution difference image LDimg is a value close to zero. On the other hand, when the difference between the temporary output image Oimg and the ideal output image is large, that is, when the difference between the temporary output image Oimg and each input image Iimg is large, The pixel value of the low resolution difference image LDimg will depart from zero. Thus, the magnitude of the deviation of the pixel value of the low resolution difference image LDimg from zero means the magnitude of the difference between the temporary output image Oimg and the ideal output image. Therefore, if the pixel value of the low resolution difference image LDimg is fed back to the temporary output image Oimg, the temporary output image Oimg can be brought close to an ideal output image.

  In the next step S150, the high resolution difference image generation unit 260 (FIG. 7) generates a high resolution difference image HDimg for each low resolution difference image LDimg. Specifically, the high-resolution difference image generation unit 260 performs high-resolution processing on the low-resolution difference image LDimg, further performs sharpness enhancement processing, and then moves the image according to the estimated relative movement amount. Each high-resolution difference image HDimg is generated. As the resolution enhancement process, various known processes can be employed. For example, a bilinear method or a nearest neighbor method can be employed. As the sharpness enhancement process, various known processes can be employed. For example, processing using an unsharp mask may be employed. Furthermore, the image moving process is performed so that the high-resolution difference image HDimg overlaps the temporary output image Oimg. At this time, each pixel value of each high-resolution difference image HDimg is corrected so as to represent a pixel value at the same pixel position as that of the temporary output image Oimg. Various known interpolation processes can be employed as the process for correcting the pixel value.

  In the next step S160, the high resolution difference image generation unit 260 combines each high resolution difference image HDimg to generate one combined high resolution difference image CHDimg. Each pixel value of the combined high-resolution difference image CHDimg is set to the sum of pixel values at the same pixel position in each high-resolution difference image HDimg.

  In the next step S170, the output high-resolution image update unit 270 feeds back the synthesized high-resolution difference image CHDimg to the temporary output image Oimg. This feedback is performed by correcting each pixel value of the temporary output image Oimg according to the following equation (1).

  Pixel value after update = Pixel value before update + α × Difference pixel value ... (1)

  Here, the updated pixel value is the pixel value of the provisional output image Oimg after feedback. The pre-update pixel value is a pixel value of the temporary output image Oimg before feedback. The difference pixel value is a pixel value at the same pixel position in the combined high-resolution difference image CHDimg. α is a coefficient representing the strength of feedback. This coefficient α may be experimentally set in advance so that feedback does not become excessive. The sign α of the coefficient α is determined so as to realize negative feedback. In the present embodiment, the difference pixel value is obtained from a value obtained by subtracting the input image Iimg from the estimated image Eimg (Eimg−Iimg), and thus the coefficient α is a negative value.

  Through the above processing, the temporary output image Oimg can be updated to an image more suitable for the input image Iimg. After step S170, the process returns to step S130, and the processes of steps S130 to S170 are repeatedly executed. Then, when the pixel value of the combined high-resolution difference image CHDimg becomes sufficiently small, the control unit 210 determines that the process has converged and completes the update. The output image Oimg finally updated in this way is used as the final output image Oimg.

  In this embodiment, since the blurring process (smoothing process) is executed in step S130, the absolute value of the pixel value of the sharp part of each input image Iimg in the low resolution difference image LDimg is the smooth part. The absolute value of the pixel value can be made larger. Furthermore, since the sharpness enhancement process is executed in step S150, it is possible to emphasize a sharp portion of each input image Iimg in the high resolution difference image HDimg. As a result, the updated temporary output image Oimg can be a sharp image that is consistent with each input image Iimg.

  Note that the composite high resolution processing shown in FIGS. 8 and 9 can be executed for the entire image, or can be executed for each line. When the process is executed for each line, at least one line image to be processed is generated as each image Eimg, LDimg, HDimg, CHDimg, and Oimg in FIG.

  Incidentally, the selection of the high resolution processing mode described with reference to FIG. 5 can be executed for each pixel or for each line as described above. When selecting the high resolution processing mode for each pixel element, for example, the composite high resolution processing shown in FIGS. 8 and 9 is executed for each line, and then the simple high resolution processing is selected. With respect to the pixel to be used, it is possible to generate a high-resolution image by using the pixel value obtained by the simple high-resolution processing instead of the pixel value obtained by the synthetic high-resolution processing.

C. Variations:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

C1. Modification 1:
In the above embodiment, the three resolution enhancement processes shown in FIG. 5 are used. However, various resolution enhancement processes other than these can also be used. Specifically, for example, in step S45a in FIG. 5, a simple high resolution process (a process that does not perform noise removal) may be selected instead of the noise removal high resolution process. In step S45b, instead of the sharpness enhancement high resolution processing, simple high resolution processing (processing that does not perform sharpness enhancement) may be selected. Further, in addition to the three resolution enhancement processes shown in FIG. 5, a simple resolution enhancement process may be added as an option. The simple high-resolution processing has an intermediate effect between the noise removal high-resolution processing and the sharpness-enhanced high-resolution processing in terms of both the high definition effect and the noise.

  As can be understood from these explanations, as the high resolution processing mode, one or more of a plurality of low resolution images are used, and a plurality of processing modes having different high definition effects are used. Is possible. Also, it is preferable to prepare three or more resolution enhancement processing modes in advance. In this way, in the obtained high-resolution image, the boundary between the positions processed in each processing mode becomes inconspicuous, so that the image quality as a whole can be improved.

C2. Modification 2:
Various processes other than those shown in FIGS. 8 and 9 can be employed as the composite high resolution process. For example, as another composite high resolution processing, low resolution pixels close to each high resolution pixel are selected one by one from the three input images Iimg1 to Iimg3, and the pixel values of these three low resolution pixel pixels are selected. A method of obtaining the pixel value of the high resolution pixel by interpolation can be employed. Also, as a pixel value interpolation calculation method, various methods such as a bilinear method and a bicubic method can be used.

C3. Modification 3:
In the above embodiment, the data processing unit 100 is provided in the printer, but instead, the data processing unit 100 may be configured using a general personal computer. The data processing unit 100 may be mounted on various other electronic devices such as an information portable terminal, a mobile phone, a digital video camera, and a digital still camera.

C4. Modification 4:
In each of the above embodiments, a high-resolution still image is generated from a moving image. Instead, a high-resolution still image may be generated from a plurality of low-resolution still images.

C5. Modification 5:
In each of the above embodiments, a part of the configuration realized by software may be replaced by hardware, and conversely, a part of the configuration realized by hardware may be replaced by software. .

1 is a block diagram illustrating a configuration of an image processing system as an embodiment of the present invention. It is a block diagram which shows the internal structure of a data processing part. It is explanatory drawing which shows an example of the relationship between the pixel position of a low resolution image and a high resolution image. It is explanatory drawing which shows the other example of the relationship between the pixel position of a low resolution image and a high resolution image. It is a flowchart which shows the whole procedure of the resolution increasing process in a present Example. It is explanatory drawing which shows the method of edge detection. It is a block diagram which shows the internal structure of a synthetic | combination high resolution process part. It is explanatory drawing which shows the content of the synthetic | combination high resolution process. It is a flowchart which shows the procedure of a synthetic | combination high resolution process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Data processing part 210 ... Control part 215 ... Image selection part 220 ... Low resolution image estimation part 230 ... High resolution image estimation part 240 ... Low resolution difference image generation part 250 ... Movement amount calculation part 260 ... High resolution difference image generation part 270 ... Output high-resolution image update unit 300 ... Digital video camera 420 ... Edge detection unit 430 ... Line buffer 440 ... High resolution processing unit 441 ... Noise removal high resolution processing unit 443 ... Sharpness enhancement high resolution processing unit 444 ... Composition High resolution processing unit 450 ... print data generation unit 460 ... motion vector detection unit 500 ... printer 700 ... print execution unit 800 ... image processing system

Claims (5)

An image processing apparatus that creates a single high-resolution image using a plurality of low-resolution images including a reference image,
An edge detection unit that detects edges in the reference image by analyzing the reference image;
A high-resolution processing unit that executes a high-resolution processing for creating one high-resolution image based on the plurality of low-resolution images;
With
The high resolution processing has a plurality of high resolution processing modes having different high definition effects as high resolution processing modes each using at least one of the plurality of low resolution images.
The high-resolution processing unit uses one high-resolution processing mode from among the plurality of high-resolution processing modes according to the size of the edge amount of the processing position that is the processing target of the high-resolution processing. An image processing apparatus that selects and executes resolution enhancement. The image processing apparatus according to claim 1,
The plurality of high resolution processing modes are:
A combined high-resolution processing mode for creating the high-resolution image by combining the plurality of low-resolution images;
A sharpness-enhancing and high-resolution processing mode for executing a simple high-resolution process and a sharpness-enhancing process for increasing the resolution of the reference image using only the reference image;
A noise removal and high resolution processing mode for performing simple high resolution processing and noise removal processing for increasing the resolution of the reference image using only the reference image;
Have
The high resolution processing unit
(I) Select and execute high resolution in the composite high resolution processing mode at a position where the edge amount is large,
(Ii) In the position where the edge amount is medium, the high resolution is selected and executed by the sharpness enhancement high resolution processing mode;
(Iii) An image processing apparatus that selects and executes high resolution in the noise removal high resolution processing mode at a position where the edge amount is small. The image processing apparatus according to claim 2,
In the sharpness high-resolution processing mode, the strength of sharpness enhancement processing is set to be larger as the edge amount is larger. An image processing method for creating a single high-resolution image using a plurality of low-resolution images including a reference image,
(A) detecting an edge in the reference image by analyzing the reference image;
(B) executing a high resolution process for creating one high resolution image based on the plurality of low resolution images;
With
The high resolution processing has a plurality of high resolution processing modes having different high definition effects as high resolution processing modes each using at least one of the plurality of low resolution images.
In the step (b), a high resolution processing mode is selected from the plurality of high resolution processing modes according to the size of the edge amount of the processing position that is the processing target of the high resolution processing. An image processing method including a step of selecting and executing resolution. A computer program for creating one high-resolution image using a plurality of low-resolution images including a reference image,
An edge detection function for detecting an edge in the reference image by analyzing the reference image;
A high resolution processing function for executing a high resolution processing for creating a single high resolution image based on the plurality of low resolution images;
Including a computer program for causing a computer to realize
The high resolution processing has a plurality of high resolution processing modes having different high definition effects as high resolution processing modes each using at least one of the plurality of low resolution images.
The high resolution processing function is based on one high resolution processing mode from among the plurality of high resolution processing modes according to the size of the edge amount of the processing position that is the processing target of the high resolution processing. A computer program that includes the function to select and execute high resolution.

Images