JP2008067110A - Generation device for superresolution image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a superresolution image generation device capable of automatically providing a superresolution image having no unevenness and maximum scale factor by using the monitor of a personal computer to make presence of unevenness be efficiently perceived or even without displaying the unevenness on the monitor of the personal computer in advance. <P>SOLUTION: The superresolution image generation device comprises: a superresolution forming part for receiving a first image having a first resolution, using the first image and generating a second image having a second resolution higher than the first resolution; a sharpness calculating part for receiving the second image generated by the superresolution forming part and calculating the sharpness of each partial area in the second image; a sharpness image generating part for receiving the sharpness of each partial area calculated by the sharpness calculating part and generating a third image having in each partial area what is converted into a value for displaying sharpness as an image; and a compositing part for receiving the first image and the third image, combining the first image with the third image and generating a fourth image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an apparatus for generating a super-resolution image.

  Sometimes you want to print a picture taken with a digital camera or video camera. For example, when printing with a resolution of 300ppi (pixel / inch) on a whole A3 paper using a personal computer and printer, the software for the personal computer, Photoshop 7.0 (Non-patent Document 1), has 4961 and 3508 pixels respectively. If you input the image of, Photoshop7.0 will help you print, so you can print easily.

  A technology to increase the number of pixels, such as a super-resolution technique, if you want to print at a resolution of 300ppi (pixel / inch) with no margins on the entire A3 paper even though the captured image does not satisfy 4961 and 3508 pixels respectively. There is a way to use.

  However, when the super-resolution technique is used, the entire image is not necessarily sharpened uniformly, and an uneven image in which blurred portions are mixed may be printed.

  FIGS. 16 and 17 show examples in which an image with an increased number of pixels is printed using the super-resolution technique. FIG. 17 shows a person in motion surrounded by an ellipse in the image printed in FIG.

As shown in FIG. 17, in an example in which an image with an increased number of pixels is printed using this super-resolution technique, the portion of the person surrounded by an ellipse is blurred, while the other portions are sharpened. It can be seen that the contrast is perceived as unevenness.

Adobe Photoshop 7.0 Japanese User Guide pp.54-61, 381-395

  As described above, even when printing is performed by increasing the number of pixels using the ultra-high resolution technique, a portion of a printed photograph that is sharpened and a portion that is not sharp may appear unevenly.

  Therefore, the user wants to check whether there is any unevenness by displaying a super-resolution image on the monitor of the personal computer in advance. In order to display an image with an increased number of pixels on a monitor of a personal computer having a resolution lower than that, it is necessary to reduce the image once and display it on the monitor. However, when a super-resolution image is reduced, the blurred portion is expressed more sharply, and there is a problem that the unevenness is not noticeable on the monitor of the personal computer and cannot be confirmed.

  FIG. 18 shows an example in which the image of FIG. 17 is reduced and displayed on the monitor. It can be seen that the blurring of the person surrounded by the ellipse in FIG. 17 is difficult to perceive in FIG.

  On the other hand, if a super-resolution image is displayed as it is on a monitor of a low-resolution personal computer without reducing it, only a part of the image can be displayed on the monitor, and the user must check the unevenness by using scrolling. .

  FIG. 19 is an example in which only the upper left portion of the image of FIG. 17 is displayed without being reduced. It can be seen that in order to check whether there is any unevenness in other parts, it is necessary to move the display position in the image and visually check whether there is unevenness in all positions while scrolling.

  The present invention has been made in order to solve the above-described problems, and it is possible to efficiently perceive the presence of unevenness using a personal computer monitor, or to display the maximum on a personal computer monitor in advance. An object of the present invention is to provide a super-resolution image generating apparatus capable of automatically providing a super-resolution image having a magnification.

In order to achieve the above object, the first invention provides:
A super-resolution processing unit that receives a first image having a first resolution and generates a second image having a second resolution higher than the first resolution by using the first image;
The second image generated by the super-resolution unit is input, and a sharpness calculation unit that calculates the sharpness of each partial region in the second image;
A sharpness image for generating a third image for which each of the partial areas has the sharpness of each partial area calculated by the sharpness calculation unit and is converted into a value for displaying the sharpness as an image. A generator,
A super-resolution comprising: a combining unit that receives the first image and the third image and generates the fourth image by combining the first image and the third image. An image generation apparatus is provided.

In addition, the second invention,
A plurality of first images having a first resolution are input, and a second image having a second resolution obtained by increasing the vertical and horizontal pixel densities from the first resolution by using the plurality of first images. A super-resolution section for generating
A plurality of the first images, and a motion vector detection unit that detects a motion vector between the plurality of first images;
A motion vector detected by the motion detector, and a pixel number calculator that counts the number of pixels present in a partial region of the position of the plurality of first images using the motion vector;
The number of pixels calculated by the number-of-pixels calculation unit and the first image are input, and a sharpness calculation unit that calculates the sharpness of each partial region of the second image using the number of pixels;
A sharpness image for generating a third image for which each of the partial areas has the sharpness of each partial area calculated by the sharpness calculation unit and is converted into a value for displaying the sharpness as an image. A generator,
A super-resolution comprising: a combining unit that receives the first image and the third image and generates the fourth image by combining the first image and the third image. An image generation apparatus is provided.

I will provide a.

In addition, the third invention,
A second resolution in which a plurality of first images having a first resolution are input, and the vertical and horizontal pixel densities are increased by M and N times, respectively, from the first resolution using the plurality of first images. A super-resolution section for generating a second image having:
A plurality of the first images are input, and a motion detection unit that detects motion between the plurality of first images;
A motion detection unit that receives motion detected by the motion detection unit and counts the number of pixels n in a partial region of the position of the plurality of first images using the motion;
The pixel number n calculated by the pixel number calculation unit, a preset threshold value, and a preset ratio of vertical and horizontal pixel density magnifications (M / N) are input, and the pixel number n is A value obtained by dividing M by the product of N (n / MN) is equal to or greater than the threshold value, and includes a pixel density magnification determining unit that determines the maximum M and N satisfying the relationship of the ratio (M / N). ,
A super-resolution image generating apparatus, wherein the super-resolution conversion unit receives the pixel density magnification determined by the pixel density magnification determination unit and generates the second image using the pixel density magnification. I will provide a.

  Since information on the sharpness is presented on the monitor of the personal computer, it is easy to understand whether there is a portion with low sharpness. Further, by using the value obtained by dividing the number of pixels necessary for super-resolution by the magnification for the sharpness, it is possible to provide an image having the maximum size within a range where no blur occurs.

  Embodiments of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to the following embodiment, It can be devised variously.

  FIG. 4 is a block diagram of a super-resolution image generating apparatus according to the embodiment of the first invention.

  As shown in FIG. 4, the super-resolution image generation device 0201 is blurred by an input unit 0202 that receives low-resolution image data 1 and a size instruction reception unit 0311 that receives size instruction data 0310 from a user. A threshold instruction receiving unit 0501 to which threshold data 0210 is input, a composite color instruction receiving unit 0502 to which the blurred region color instruction data 0212 is input, and low resolution image data 1 Super-resolution processing unit 0203 for generating image data 2, sharpness calculation unit 0204 for calculating sharpness data 0205 for each partial region in high-resolution image data 2, and sharpness data 0205 based on threshold data 0503 A sharpness image generation unit 0206 for generating sharpness image data 3 for displaying an image for each image, and switching instruction data 211, a switching unit 0207 for switching between the low resolution image data 1 and the high resolution image data 2, a synthesis unit 0208 for synthesizing the low resolution image data 1 and the sharpness image data 3, and display image data 4 from the synthesis unit 0208. And a display unit 0209 for displaying.

  Next, the operation of the super-resolution image generation device 0201 will be described.

  First, the low resolution image data 1 is input to the input unit 0202 from the outside of the super resolution image generation device 0201. Further, the size instruction data 0310 is input to the size instruction receiving unit 0311, and the threshold value instruction data 0210 is input to the threshold value instruction receiving unit 0501. Also, the switching instruction data 0211 is input to the switching unit 0207, and the composite color instruction data 0212 is input to the composite color instruction receiving unit 0502.

  Here, the image data is data having the number of pixels in the vertical direction and the horizontal direction of the image, and a value representing light and shade or color in each pixel. For example, grayscale image data having a height of H and a width of W can be defined by a matrix of H, W, and H rows and W columns. The size instruction data 0310 is data representing the height and width of the high-resolution image data 2 in which the resolution generated by the super-resolution processing unit 0203 described later is increased.

  Next, the input unit 0202 outputs the low-resolution image data 1 as it is and sends it to the super-resolution unit 0203 and the switching unit 0207.

  The size instruction receiving unit 0311 outputs the size instruction data 0310 as it is and sends it to the super-resolution unit 0203.

  Next, the threshold value instruction receiving unit 0501 sets and outputs threshold data 0503 in accordance with the threshold value instruction data 0210, and sends it to the sharpness image generation unit 0206. The threshold instruction receiving unit 0501 is, for example, a slide bar, and the threshold instruction data 0210 is data representing the position of the slide bar. At this time, by setting the minimum value and the maximum value of the threshold value in advance, the threshold value data 0503 is calculated and set from the position of the slide bar.

  Further, when the threshold value data 0503 is changed by the threshold value instruction data 0210, the display image data 4 to be described later can be displayed in conjunction with it, so that which part has low sharpness and which part It becomes easy to understand whether the price is high. Here, the sharpness is a value indicating how sharply a certain subject is moving.

  In the composite color instruction receiving unit 0502, the composite color data 0504 is set and output in accordance with the composite color instruction data 0212, and sent to the combining unit 0208. The composite color instruction receiving unit 0502 is, for example, a color pallet, and the composite color instruction data 0212 is data representing a position in the color pallet. The composite color data 0504 is set as a color corresponding to the position.

  Further, when the composite color data 0504 is changed by the composite color instruction data 0212, the display image data 4 described later can be displayed in conjunction with the display image data 4 so that the display image data 4 is combined with the composite color preferred by the user. The image will be displayed. If the composite color data 0504 is fixed with a predetermined color, the composite color instruction data 0212 and the composite color instruction reception unit 0502 are not necessary.

  The super-resolution unit 0203 generates high-resolution image data 2 in which the low-resolution image data 1 is super-resolved to a height H and a width W represented by the size instruction data 0310. The super-resolution unit 0203 can use a super-resolution technique such as increasing the number of pixels based on a plurality of input images, or increasing the number of pixels using image similarity. Next, the high-resolution image data 2 generated by the super-resolution unit 0203 is sent to the sharpness calculation unit 0204 and the switching unit 0207.

  The sharpness calculation unit 0204 calculates the sharpness for each partial area of the high-resolution image data 2, and a set of sharpnesses for each partial area is output as the sharpness data 0205 and sent to the sharpness image generation unit 0206. It is done.

  Here, the partial area is, for example, an area of A and B pixels in the vertical and horizontal directions. If there are C and D partial areas in the high resolution image data 2, the sharpness data 0205 can be expressed by a matrix of C rows and D columns. Each element of the matrix is the sharpness of each partial area. The sharpness of a certain partial area is determined by, for example, a value obtained by applying a higher weight to a coefficient corresponding to a higher frequency among coefficients obtained by performing two-dimensional discrete Fourier transform on the gray value of a pixel in the partial area. Can be defined. Alternatively, the value can be defined by a value obtained by dividing the value by the number of pixels in the partial area.

  In these definitions, the sharpness increases as the partial region has a higher frequency component. When A and B are raised to powers of 2 such as A = B = 8, it is convenient for calculation of a two-dimensional discrete Fourier transform. A and B may be set by the user.

  Next, the sharpness image generation unit 0206 generates sharpness image data 3 in which the sharpness data 0205 has 1 in a partial area smaller than the threshold data 0503 and 0 in other areas, and is sent to the synthesis unit 0208. .

  FIG. 7 shows an example in which the high resolution image data 2 is divided into partial areas without overlapping. Here, a circle in FIG. 7 represents a pixel, and a square represents a partial region. In this example, the pixel number width W = 24, the pixel number length H = 16, the partial region pixel number width A = 8, the partial region pixel number length B = 8, the partial region number length C = 2, and the partial region number width D = 3.

  Further, FIG. 8 shows low resolution image data 1 in which the number of pixels and the pixel density are 1/4 with respect to the high resolution image data 2 in FIG. 7 and a partial area corresponding to the partial area described in FIG. In this example, the pixel number width W = 12, the pixel number length H = 8, the partial region pixel number width A = 4, the partial region pixel number length B = 4, the partial region number length C = 2, the partial region number width D = 3.

  Next, in the switching unit 0207, the low resolution image data 1 and the high resolution image data 2 are switched by the switching instruction data 0211, and one of them is sent to the synthesis unit 0208.

  In the combining unit 0208, the region corresponding to the region where the sharpness image data 3 is 1 in the low resolution image data 1 or the high resolution image data 2 is combined with the composite color data 0504 in a translucent manner, and the remaining regions are low. Display image data 4 that is the same as the resolution image data 1 or the high resolution image data 2 is generated and sent to the display unit 0209. In the display unit 0209, an image represented by the display image data 4 is displayed.

  FIG. 15 shows an example in which the image represented by the display image data 4 when the composite color data 0504 is red (represented by shading in FIG. 15) is displayed on the computer monitor.

  Next, the processing flow of the super-resolution image generating apparatus 0201 related to the first invention will be described with reference to FIG.

  First, as shown in FIG. 1, low resolution image data 1 is input to the input unit 0202 (step S0101).

  Next, the low resolution image data 1 is subjected to the super resolution processing in the super resolution processing unit 0203, and the high resolution image data 2 having an increased resolution is generated (step S0102).

  Next, the sharpness calculation unit 0204 calculates sharpness data 0205 that is a set of sharpness for each partial region of the high-resolution image data 2 (step S0103).

  Next, the sharpness image generation unit 0206 generates sharpness image data 3 in which the sharpness data 0205 has 1 in a partial region smaller than the threshold data 0503 and 0 in other regions (step S0104).

  Next, in the low-resolution image data 1 or the high-resolution image data 2, the region corresponding to the region where the sharpness image data 3 is 1 is combined with the composite color 0504 in a translucent manner, and the remaining region is the low-resolution image data. Display image data 4 that is the same as 1 or the high-resolution image data 2 is generated (step S0105).

  Next, the display image data 4 is displayed on the monitor of the computer (step S0106).

  In this way, as described with reference to FIG. 15, the partial area in which the sharpness data 0205 is smaller than the threshold data 0503 is also displayed on the monitor of the computer having a low resolution by being semitransparently combined with the combined color data 0504 and displayed. In this way, a mixture of a low sharpness portion and a high sharpness portion can be seen at a glance on a computer monitor. Therefore, the presence or absence of unevenness can be recognized at a glance, and it becomes easy to generate a super-resolution image without unevenness.

  In the above-described embodiment, the low-resolution image data 1 has been described by taking a grayscale image as an example. However, a color image may be used. In that case, the sharpness data 0205 is calculated after converting the color image into a grayscale image, or the average of the sharpness calculated for each color component is redefined as the sharpness data 0205. This makes it possible to apply not only to grayscale images but also to color images.

  In this embodiment, as an example of a method for calculating the sharpness data 0205, a method using two-dimensional discrete Fourier transform is described. However, other calculation methods may be used.

  Next, a super-resolution image generating apparatus according to the second invention will be described with reference to the block diagram of FIG. This super-resolution image generating apparatus generates a high-resolution image using a plurality of low-resolution image data 1, and the overlap between the plurality of low-resolution image data is the pixel density magnification due to the super-resolution. On the other hand, the higher the resolution, the higher the sharpness of the high-resolution image data, and the sharpness is obtained by dividing the value representing the degree of overlap between the low-resolution image data by the pixel density magnification due to the super-resolution. The degree data is calculated.

  As shown in FIG. 5, this super-resolution image generation device 0201 uses an input unit 0202 to which a plurality of low-resolution image data 1 are input and a plurality of low-resolution image data 1 to increase the vertical and horizontal pixel densities M times. And a super-resolution processing unit 0203 that generates high-resolution image data 2 increased N times, and a motion detection unit 0301 that receives a plurality of low-resolution image data 1 and detects motion among the plurality of low-resolution image data 1 The motion data 0302 detected by the motion detection unit 0301 is input, and the number n of pixels existing in the partial area of the combination of the positions of the plurality of low-resolution image data is counted using the motion data 0302. Pixel number calculation unit 0303 for calculating pixel number data 0304 given a set of number n, and pixel number data 030 calculated by pixel number calculation unit 0303 4 and the low-resolution image data 1 are input, and the sharpness calculation unit 0204 that calculates the sharpness data 0205 for each partial region of the low-resolution image data 1 using the pixel number n data 0304, and the sharpness calculation unit 0204 The sharpness data 0205 for each partial area is input, the sharpness image generation unit 0206 for generating the sharpness image data 3 for displaying the sharpness data 0205 for each partial area, the low-resolution image data 1 and the sharpness. The image data 3 is input and the low-resolution image data 1 and the sharpness image data 3 are combined to generate the display image data 4. The display image data 4 combined by the combining unit 0208 is input and the display image data 4 is input. And a display unit 0209 for displaying data 4.

  Similarly to the first invention, a size instruction receiving unit 0311 that receives a size instruction from a user, a threshold instruction receiving unit 0501 that receives threshold data 0210 that is a criterion for determining whether or not the image is blurred, and a blur A composite color instruction receiving unit 0502 for instructing the color of the area and a super resolution instruction receiving unit 0313 for inputting super resolution instruction data and instructing the super resolution to the super resolution unit 0203 are provided.

  The input unit 0202 outputs a plurality of low-resolution image data 1 and sends the low-resolution image data 1 to the motion detection unit 0301, the pixel number calculation unit 0303, and the sharpness calculation unit 0204.

  In the motion detection unit 0301, motion between images included in the plurality of low resolution image data 1 is detected with an accuracy finer than the pixel interval of the low resolution image data 1, and is output as motion data 0302, to the pixel number calculation unit 0303. Sent. The motion data 0302 is obtained as a motion of the entire image, is obtained as a motion for each partial area in the image, or is obtained as a motion of each pixel.

  In the pixel number calculation unit 0303, the height and width of the low resolution image data 1 and the motion data 0302 are input, and the low resolution image data is based on one of the images included in the plurality of low resolution image data 1 from these. The position of the image in 1 is adjusted. Then, the number of pixels n in the region at the same position as the partial region of the generated high-resolution image data 2 is counted, and a set of n for all the partial regions is sent to the sharpness calculation unit 0204 as pixel number data 0304.

  A method of calculating the pixel number data will be described with reference to FIGS.

  9 and 11 show the low-resolution image data 1, and FIG. 10 shows the high-resolution image data 2 in which the number of pixels of the low-resolution image data 1 is increased three times vertically and horizontally. A circle in the figure represents one pixel.

  FIG. 12 shows how the low-resolution image data 1 shown in FIGS. 9 and 11 are aligned.

  FIG. 13 represents a partial area in which pixels of the high-resolution image data 2 of FIG.

  14 represents the partial area at the same position as the partial area shown in FIG. 13 with a quadrilateral for the aligned low-resolution image data 1 shown in FIG. Since there are two pixels in the square in FIG. 14 in this way, the number of pixels n in the partial area at the same position as the partial area in FIG.

  In the sharpness calculation unit 0204, the sharpness in each partial area is calculated by dividing the number n of pixels in the partial area in the pixel number data 0304 by the magnification m of the pixel density due to the super-resolution. As in the previous example, let us consider a case where the high-resolution image data 2 of FIG. 10 is generated from the low-resolution image data 1 whose pixel positions are as shown in FIG. In this case, since the number of pixels increases (3 × 3 =) 9 times and the magnification of the pixel density is 9, m = 9. The pixel number n of the partial area in the pixel number data 0304 was 2. Therefore, the value of 2/9 obtained by dividing n by m is the sharpness of the partial region in FIG.

  Similarly, the data obtained by calculating the sharpness for the other partial regions and outputting them are output as the sharpness data 0205.

  As described above, the sharpness calculation unit 0204 calculates the sharpness in each partial region of the high-resolution image data 2 to be generated from the height and width of the low-resolution image data 1, the size instruction data 0310, and the pixel number data 0304. A set of sharpness n / m for all partial regions is sent to the sharpness image generation unit 0206 as sharpness data 0205.

  This definition does not require super-resolution processing for sharpness calculation. Therefore, the sharpness can be calculated before generating the super-resolution image. Accordingly, the time until the display image data 4 is displayed can be shortened. Since the time is shortened, the time from when the user changes m by changing the size instruction data 0310 to when the display image data 4 is displayed is shortened. Since the time until display is shortened, the stress on the user can be reduced.

  Pixels with unreliable motion detection accuracy may not be counted as pixels for sharpness calculation. In addition, the definition may be changed so that a pixel with reliable motion detection accuracy is given a greater weight and counted.

  Next, the processing flow of the super-resolution image generating apparatus 0201 related to the second invention will be described with reference to FIG.

  First, as shown in FIG. 2, a plurality of first image data (low-resolution image data) 1 is input to the input unit 0202 (step S0301).

  Next, a plurality of low resolution image data 1 is input to the motion detection unit 0301 from the input unit 0202, and motion data 0302 is calculated from the plurality of low resolution image data 1 (step S0302).

  Next, the pixel number calculation unit 0303 aligns the positions of the images in the low-resolution image data 1, and pixel number data 0304, which is a set of the number of pixels corresponding to each partial area of the display image data 4 to be generated, is obtained. It is obtained (step S0303).

  Next, the sharpness calculation unit 0204 calculates the sharpness data 0205, which is a set of sharpness for each partial region corresponding to the high resolution image data 2 using the pixel number data 0304 (step S0304).

  Next, the sharpness image generation unit 0206 generates sharpness image data 3 in which the sharpness data 0205 has 1 in a partial area smaller than the threshold data 0503 and 0 in other areas (step S0305).

  Next, in the combining unit 0208, the region corresponding to the region where the sharpness image data 3 is 1 is combined with the combined color data 0504 in a translucent manner, and the remaining region is the same as the first image data. Data 4 is generated (step S0306).

  Next, the display image data 4 is displayed on the monitor of the computer (step S0307).

  When the super-resolution instruction data 0312 is input to the super-resolution section 0203, at least one of the low-resolution image data 1 is represented by the size instruction data 0310 from the low-resolution image data 1 and the motion data 0302. High-resolution image data 2 that is super-resolved in width and height is generated.

  As one of the super-resolution techniques, from the set of learning low-resolution images and high-resolution images, the correspondence relationship is registered in a database for each partial area, and the super-resolution processing is performed using the database. There is something. In this method, the super-resolution processing is performed for each partial region.

  In this method, first, a partial region of a low resolution image similar to a partial region of a low resolution image to be super-resolution is found that has a small distance from the partial region of the low resolution image in the database registered at the time of learning. Search by. Then, a super-resolution image is generated by using the feature of the partial region of the high-resolution image corresponding to it.

  For example, the corresponding partial area of the high-resolution image may be employed as it is as the partial area of the super-resolution image. In some cases, the high-resolution component of the corresponding partial region of the high-resolution image is added to the enlarged low-resolution image to be super-resolution by interpolation. In any case, there is a property that a sharper super-resolution image is generated as the distance d of the partial area of the low-resolution image searched from the database to the partial area of the low-resolution image desired to be super-resolution is closer.

  Therefore, using this property, 1 / (d + ε) can be used as the sharpness. Here, ε is a small positive number close to 0, and is a number for preventing division by 0 when d is 0. By doing so, it is possible to define so that what is searched from the database has a greater sharpness as it becomes closer to the super resolution. In this method, the sharpness can be calculated as long as the distance can be calculated. Therefore, it can be calculated before the super-resolution image is generated.

  Therefore, there is an effect that the time from when the user changes m to when the display image data 4 is displayed can be shortened.

  In this embodiment, the sharpness is calculated for all partial areas of the super-resolution image. However, since it is difficult to perceive blur in the flat partial area, the sharpness calculation may be omitted. Along with this change, it is preferable to change so that information regarding the sharpness is not presented in the partial region where the calculation of the sharpness is omitted. The flat partial region may be defined as a partial region that has been subjected to edge detection on the low-resolution image data 1 or the super-resolution image and determined not to have an edge. Thereby, the amount of calculation can be reduced.

  In addition, as a sharpness calculation method, for example, a difference between delta histograms respectively obtained from a partial area of a super-resolution image and an image subjected to an average value filter can be used as the sharpness.

  Here, a method for obtaining a delta histogram from a partial region of an image will be described.

  First, the gray value of a certain pixel in the partial region of the image and the absolute value of the difference between the gray values of the pixels adjacent to the upper left, upper, upper right, left, right, lower left, lower and lower right of the pixel are obtained. In this way, eight values are obtained.

  Next, these eight values are obtained in the same manner for other pixels in the partial region. Then, 8E values which are the number obtained by multiplying the number E of pixels other than the outer periphery in the partial region by 8 are obtained. Finally, a histogram having the absolute value of the difference on the horizontal axis and the frequency on the vertical axis is obtained from the 8E values. Here, in the case of a grayscale image of 8 gradations, the absolute value of the difference takes a value from 0 to 255. Thus, a delta histogram of the partial area of the image is obtained.

The difference between the delta histograms H (1) and H (2) is, for example,

Calculate with Here, max (·, ·) is a function that takes the maximum value of two values, and min (·, ·) is a function that takes the minimum value.

  Further, the difference between the accumulated delta histograms obtained from the partial area of the super-resolution image and the image subjected to the average value filter may be used as the sharpness.

  The cumulative delta histogram is obtained by making the vertical axis the cumulative frequency from the delta histogram obtained by the above method. Others can be calculated in the same way.

  Also, the sharpness of the partial area of the super-resolution image is calculated, the sharpness of the area of the low-resolution image data 1 corresponding to the partial area is further calculated, and the value obtained by dividing the former by the latter is used again as the sharpness. Can do.

  By doing so, the sharpness of the portion blurred by the super-resolution processing is reduced, so that the position of unevenness generated by the super-resolution processing becomes easy to understand.

  In addition, in the super-resolution image, the vicinity of a partial area with low sharpness can be displayed on the monitor of the personal computer. This makes it easy to see how much the low sharpness portion is blurred. Of course, the user may be allowed to select which part of the super-resolution image is displayed. Thereby, it becomes easy to confirm whether or not the part that the user particularly wants to pay attention is not blurred.

  It is also possible to display the image after changing the image size so that the entire super-resolution image fits on the display unit 0209. This makes it easier to detect when unnatural noise occurs due to super-resolution.

  In the present embodiment, only the portion having sharpness lower than the threshold value data 0503 is semi-transparently synthesized with the synthesized color data 0504, but may be synthesized by another method.

  For example, only a portion having a higher sharpness than the threshold data 0503 may be combined with the composite color data 0504 in a translucent manner. Even if it changes in this way, it can confirm at a glance that a low sharpness part and a high part are mixed.

  Further, without using the threshold value data 0503, a transparency called an alpha value can be set in proportion to or in inverse proportion to the value of the sharpness data 0205, so that it can be semi-transparently synthesized with the synthesized color data 0504. As a result, the higher the sharpness or the lower the sharpness, the deeper the composite color 0504 is synthesized. Along with this change, the threshold instruction receiving unit 0501, the threshold instruction data 0210, and the threshold data 0503 become unnecessary.

  In the present embodiment, the entire low-resolution image data 1 is made super-resolution, but only a part of it can be saved or printed. It was difficult to select the area to be saved or printed because it was difficult to see the unevenness even when the super-resolution image was reduced. On the other hand, by designating an area from the displayed display image data 4 and saving or printing only the area in the super-resolution image corresponding to the area, it becomes easy to select a non-uniform area. As a result, it becomes easy to store and print a uniform super-resolution image.

  In the present embodiment, the entire low-resolution image data 1 is converted into the super-resolution, but only a part of the low-resolution image data 1 can be converted into the super-resolution. When the sharpness can be calculated before the super-resolution, the display image data 4 can be generated and displayed before the super-resolution. If the user can select a region to be super-resolution based on the displayed display image data 4, it is easy to generate a uniform super-resolution image.

  Further, the paper size suitable for printing the super-resolution image may be changed to be presented. If the printing area in the super-resolution image and the resolution at the time of printing are determined, paper sizes such as A3 and A4 suitable for the printing area can be calculated. As the print area, an area of the super-resolution image corresponding to the displayed display image data 4 is set by designating the area. The resolution is automatically determined from the performance of the printer, or is determined by the image quality required by the photograph.

  It is preferable to automatically determine the print direction of whether to print vertically or horizontally from the height and width of the print area. As a result, the user does not have to specify whether the user is vertically or horizontally.

  If the paper size, printing direction, and resolution are determined, the number of vertical and horizontal pixels necessary for printing can be calculated. Using this property, the user can easily specify the print area by setting the paper size, print direction, and resolution. Specifically, if a rectangle having the calculated number of pixels is displayed and the user can move it, the print area can be easily specified. If the rectangle is displayed on the display image data 4, the sharpness is high and it is easy to designate a print area having a field angle that the user likes.

  Further, the super-resolution unit 0203 can widen the angle of view of the super-resolution image by connecting a plurality of low-resolution image data 1 to each other and further super-resolution the data.

  FIG. 20 is a block diagram of the super-resolution image generating apparatus 0201.

  In the configuration of FIG. 5, a joining unit 2001, a region instruction receiving unit 2004, and a magnification instruction receiving unit 2007 are added, and a size instruction receiving unit 0311 is deleted. The same parts as those in FIG. 5 are denoted by the same reference numerals, and the description thereof is omitted.

  The input unit 0202 outputs the low-resolution image data 1 as it is and sends it to the motion detection unit 0301 and the joining unit 2001.

  In the joint unit 2001, image data 5 in which images in the low resolution image data 1 are joined is generated from the motion data 0302 and the low resolution image data 1, and is sent to the pixel number calculation unit 0303.

  These processes will be described with reference to FIG. 21 to FIG.

  First, in the landscape shown in FIG. 21, the part of FIG. 22 (a) is photographed at time t, FIG. 22 (b) is photographed at time t + 1, and FIG. 22 (c) is photographed at time t + 2. think of. As described above, when the low resolution image data 1 is the three pieces as shown in FIG. 22, the joining unit 2001 performs alignment based on the motion data 0302 to generate the image data 5.

  FIG. 23 shows the connected image data 5.

  FIG. 24 shows the positions of the pixels of the connected image data 5. A round shape in the figure represents a pixel. Thus, the image data 5 holds the pixel position.

  In the pixel number calculation unit 0303, the number n of pixels in each partial region is counted from the image data 5, and a set of these is output as pixel number data 0304 and sent to the sharpness calculation unit 0204.

  In the magnification instruction receiving unit 2007, the pixel density magnification 2008 is determined from the magnification instruction data 2006 input from the outside of the super-resolution image generation device 0201. This pixel density magnification 2008 is the same as m described above. The magnification instruction receiving unit 2007 is a numeric keypad, for example, and the magnification instruction 2006 is a key input by the user.

  In the sharpness calculation unit 0204, the sharpness in each partial area of the generated super-resolution image is calculated from the pixel density magnification 2008 and the pixel number data 0304, and a set of sharpness for all partial areas is used as the sharpness data 205. It is output and sent to the sharpness image generation unit 0206.

  In the area instruction receiving unit 2004, area data 2005 is determined from the area instruction data 2003 input from the outside of the super-resolution image generating device 0201 and sent to the synthesizing unit 0208.

  The area instruction receiving unit 2004 is an input interface such as a mouse, for example, and the area instruction data 2003 is an instruction by a user who uses the interface. Area data 2005 represents an area determined by the instruction.

  FIG. 25 shows an example of the area data 2005 when the area is rectangular.

  In the combining unit 0208, the outline of the area data 2005 combined with a broken line is sent as display image data 4 to the display unit 0209. If the area data 2005 is not designated, the broken line is not synthesized. Alternatively, only the inside of the area data 2005 may be output as the display image data 4. Then, when the image is displayed, the user can easily understand what the angle of view is.

  FIG. 26 is a flowchart showing the flow of processing in this case.

  In steps S0601 and S0602 in FIG. 26, the same processing as in steps S0301 and S0302 described in FIG. 2 is performed.

  Next, the images in the low-resolution image data 1 are connected based on the motion to generate image data 5 (step S0603).

  Next, by counting the number of pixels in the partial area of the image data 5, pixel number data 0304 is generated (step S0604).

  Next, the sharpness data 205 is calculated (step S0605).

  Next, in step S0606, the same processing as in step S0305 described in FIG. 2 is performed.

  Next, display image data 4 is generated (step S0607).

  Next, in step S0608, the same processing as step S0307 described in FIG. 2 is performed.

  The images connected by performing alignment based on the movement in this way have a height and width that are larger than those that are not connected. When the connected images are made super-resolution, the height and width are further increased. When the super-resolution image is reduced and displayed on the display device so as to fit the whole image, unevenness becomes even more difficult to understand. Here, regardless of the reduction ratio, the region with low sharpness is combined with the combined color data 0504 in a translucent manner, so that it is easy to understand the presence or absence of unevenness.

  Furthermore, since the region 2005 is combined and displayed with a broken line, it is easy to understand which portion of the display image data 4 is printed with a super-resolution so that unevenness does not occur. Further, since the obtained movement is used for joining images and further used for obtaining the sharpness, there is no waste in processing.

  FIG. 6 is a block diagram of a super-resolution image generating apparatus according to the embodiment of the third invention.

  As shown in FIG. 6, this super-resolution image generation device 0201 has an input unit 0202 to which a plurality of low-resolution image data 1 are input, and a vertical direction and a horizontal direction at the time of super-resolution by inputting ratio instruction data 0801. A ratio instruction receiving unit 0802 for setting ratio data 0803, which is a ratio (M / N) of pixel density magnification, and a plurality of low resolution image data 1 are used to increase vertical and horizontal pixel densities to M times and N times, respectively. A plurality of low-resolution image data 0203, a low-resolution image data 1 are input, a motion detection unit 0301 that detects motion between the plurality of low-resolution image data 1, and a motion detected by the motion detection unit 0301 are input. A pixel number calculation unit 0303 that counts the number of pixels n present in a partial region of a combination of positions of a plurality of image data using motion, and threshold value instruction data 0210 are input and threshold data Threshold value reception unit 0501 for setting 0503, pixel number n data 0304 calculated by pixel number calculation unit 0303, threshold value data 0503, and ratio data 0803 are input, and pixel number n data 0304 is divided by the product of M and N The pixel density magnification determination unit 0601 determines pixel density magnification data 0602 representing the maximum M and N satisfying the relationship of the ratio data 0803 with the value (n / MN) being the threshold data 0503 or more.

  The input unit 0202 outputs the low-resolution image data 1 as it is and sends it to the motion detection unit 0301 and the pixel number calculation unit 0303.

  Further, in the processing in the pixel number calculation unit 0303, the pixel number data 0303 is sent to the pixel density magnification determination unit 0601. Further, the ratio instruction accepting unit 0802 is formed of, for example, a numeric keypad, and a key input string pressed by the user is input as the ratio instruction data 0801, which is converted into a numerical value and output as a ratio 0803, and the pixel density magnification It is sent to the determination unit 0601.

  Further, the pixel density magnification determining unit 0601 obtains the pixel density magnification M in the vertical direction and the pixel density magnification N in the horizontal direction from the pixel number data 0304, the threshold data 0503, and the ratio data 0803, respectively. Is output as pixel density magnification data 0602.

  Further, the super-resolution unit 0203 superimposes at least one of the low-resolution image data 1 from the low-resolution image data 1, the motion data 0302, and the pixel density magnification data 0602 at the pixel density magnification represented by the pixel density magnification data 0602. High-resolution image data 2 having a resolution is generated.

  Next, a method for calculating M and N in the pixel density magnification determination unit 0601 will be described. Here, it is assumed that the sharpness is defined by n / m. Since the magnification m of the pixel density is equal to MN, which is the product of M and N, increasing M and N increases m. Therefore, when M and N are increased, the sharpness n / m is decreased.

  On the other hand, since the pixel number n data depends only on the partial region of the super-resolution image for which the sharpness is calculated, it does not change even if M or N is increased. Therefore, by dividing n_min by the threshold value data 0503, m can be obtained in which the sharpness is equal to or higher than the threshold value data 0503 in all partial regions. Here, n_min is the minimum sharpness in the sharpness data 0205.

When the ratio of M and N is represented by M = aN, since m = MN, M> 0, and N> 0, M and N at which the sharpness in all partial regions is greater than or equal to the threshold value 0503 are

Can be determined. Here, a represents the ratio data 0803, and T represents the threshold data 0503.

  Here, the ratio data 0803, that is, a is given by the user, but a is not necessarily given by the user. For example, if the ratio of the height and width of the high-resolution image 2 to be generated is determined in synchronization with the display device or printing paper, a may be automatically set according to the ratio. Alternatively, a may be set to 1 if the same aspect ratio as that of the original low-resolution image is desired.

  The flow of processing is shown in FIG.

  First, a plurality of low resolution image data 1 is input to the input unit 0202 (step S0401).

  Next, the motion detection unit 0301 obtains motion data between the plurality of low resolution image data 1 (step S0402).

  Next, the number of pixels is calculated from the motion data 0302 from the motion detector 0301 and the plurality of low-resolution image data 1 (step S0403).

  Next, the threshold value instruction receiving unit 0510 to which the threshold value instruction data 0210 has been input sets the threshold value data 0503 (step S0404).

  Next, the ratio instruction receiving unit 0802 sets ratio data 0803 (a) that is a ratio of M and N (step S0405).

  Next, the pixel density magnification determining unit 0601 determines pixel density magnification data 0602 (M and N) (step S0406).

  As described above, the maximum M and N that have the sharpness n / m equal to or higher than the threshold value data 0503 (T) and satisfy the set ratio are automatically determined in all partial areas.

It is a flowchart showing the flow of a process of the production | generation apparatus of the super-resolution image in connection with 1st Embodiment of this invention. It is a flowchart showing the flow of a process of the production | generation method of the super-resolution image in connection with 2nd Embodiment of this invention. It is a flowchart showing the flow of a process of another modified example of the production | generation assistance method of the super-resolution image in connection with embodiment of 3rd invention of this invention. It is a block diagram showing the structure of the production | generation assistance apparatus of the super-resolution image in connection with embodiment of 1st invention of this invention. It is a block diagram showing the structure of the example of a change of the production | generation assistance apparatus of the super-resolution image in connection with embodiment of 2nd invention of this invention. It is a block diagram showing the structure of another example of a change of the super-resolution image production | generation assistance apparatus in connection with embodiment of 3rd invention of this invention. It is a figure for demonstrating a super-resolution image and its partial area | region. It is a figure for demonstrating the low resolution image corresponding to FIG. 7, and the partial area of the same position as the partial area of FIG. It is a figure showing the example of a low resolution image. FIG. 10 is a diagram illustrating a super resolution image in which the number of pixels of the low resolution image in FIG. FIG. 10 is a diagram illustrating a low-resolution image different from that of FIG. 9 used for generating the super-resolution image of FIG. 9. It is a figure which shows a mode that the position of the low resolution image of FIG. 9 and FIG. 11 was match | combined. It is a figure for demonstrating a certain partial area | region among the super-resolution images of FIG. FIG. 11 is a diagram for explaining a partial region at the same position as the partial region in FIG. 10 in the combination of the positions of the low-resolution images in FIG. 9 and FIG. 11. An example in which an image represented by the display image data 4 when the composite color 0504 is red (displayed with diagonal lines) is displayed. It is a figure which shows a mode that the image with unevenness was printed from the printer. It is the figure which surrounded the blurred part of the image of FIG. 16 with the ellipse. It is a figure which shows a mode that the image of FIG. 16 was reduced and displayed on the monitor. It is a figure which shows a mode that it displayed on the monitor, without reducing only a part of image of FIG. It is a block diagram showing the structure of the production | generation apparatus of the super-resolution image in connection with another embodiment of the 2nd invention of this invention. It is a figure showing scenery. It is a figure showing a mode that the scenery of FIG. 21 was image | photographed at each time. It is a figure showing a mode that the picked-up image of FIG. 22 was joined. It is a figure showing the position of the pixel of the image of FIG. It is a figure showing the area | region determined by the user's instruction | indication. It is a flowchart showing the flow of a process of the production | generation apparatus of the super-resolution image in connection with another embodiment of 2nd invention of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Low-resolution image data 2 ... High-resolution image data 3 ... Sharpness image data 4 ... Display image data 0201 ... Super-resolution image generation apparatus 0202 ... Input part 0203 ... Super resolution unit 0204 ... sharpness calculation unit 0205 ... sharpness data 0206 ... sharpness image generation unit 0207 ... switching unit 0208 ... synthesis unit 0209 ... display unit 0210 ... Threshold value instruction data 0211 ... Switching data 0212 ... Composite color instruction data 0310 ... Size instruction data 0311 ... Size instruction reception unit 0501 ... Threshold instruction reception unit 0502 ... Composite color instruction reception unit 0503 ... Threshold data 0504 ... Composite color data

Claims (8)

A super-resolution processing unit that receives a first image having a first resolution and generates a second image having a second resolution higher than the first resolution by using the first image;
The second image generated by the super-resolution unit is input, and a sharpness calculation unit that calculates the sharpness of each partial region in the second image;
A sharpness image for generating a third image for which each of the partial areas has the sharpness of each partial area calculated by the sharpness calculation unit and is converted into a value for displaying the sharpness as an image. A generator,
A super-resolution comprising: a combining unit that receives the first image and the third image and generates the fourth image by combining the first image and the third image. Image generation device. A plurality of first images having a first resolution are input, and a second image having a second resolution obtained by increasing the vertical and horizontal pixel densities from the first resolution by using the plurality of first images. A super-resolution section for generating
A plurality of the first images, and a motion vector detection unit that detects a motion vector between the plurality of first images;
A motion vector detected by the motion detector, and a pixel number calculator that counts the number of pixels present in a partial region of the position of the plurality of first images using the motion vector;
The number of pixels calculated by the number-of-pixels calculation unit and the first image are input, and a sharpness calculation unit that calculates the sharpness of each partial region of the second image using the number of pixels;
A sharpness image for generating a third image for which each of the partial areas has the sharpness of each partial area calculated by the sharpness calculation unit and is converted into a value for displaying the sharpness as an image. A generator,
A super-resolution comprising: a combining unit that receives the first image and the third image and generates the fourth image by combining the first image and the third image. Image generation device. A second resolution in which a plurality of first images having a first resolution are input, and the vertical and horizontal pixel densities are increased by M and N times, respectively, from the first resolution using the plurality of first images. A super-resolution section for generating a second image having:
A plurality of the first images are input, and a motion detection unit that detects motion between the plurality of first images;
A motion detection unit that receives motion detected by the motion detection unit and counts the number of pixels n in a partial region of the position of the plurality of first images using the motion;
The pixel number n calculated by the pixel number calculation unit, a preset threshold value, and a preset ratio of vertical and horizontal pixel density magnifications (M / N) are input, and the pixel number n is A value obtained by dividing M by the product of N (n / MN) is equal to or greater than the threshold value, and includes a pixel density magnification determining unit that determines the maximum M and N satisfying the relationship of the ratio (M / N). ,
A super-resolution image generating apparatus, wherein the super-resolution conversion unit receives the pixel density magnification determined by the pixel density magnification determination unit and generates the second image using the pixel density magnification. .   The super-resolution image generating apparatus according to claim 1, wherein the synthesizing unit synthesizes the first image and the third image translucently.   The sharpness calculation unit calculates the sharpness by adding a higher weight to a coefficient corresponding to a higher frequency among coefficients obtained by performing a two-dimensional discrete Fourier transform on the partial region of the second image. The super-resolution image generating apparatus according to claim 1.   The sharpness calculation unit calculates a delta histogram from each of the second image and a partial region of an image obtained by applying an average value filter to the second image, and calculates a difference between the respective delta histograms. The apparatus for generating a super-resolution image according to claim 1, wherein sharpness is calculated.   3. The super-resolution generation apparatus according to claim 2, wherein the sharpness calculation unit calculates the sharpness by calculating a value obtained by dividing the number of pixels n by a pixel density magnification.   The supervision system according to claim 1, further comprising a display unit that receives the fourth image synthesized by the synthesis unit and displays the fourth image. A resolution image generator.

Images