JP2009093676A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a high resolution image at high speed. <P>SOLUTION: An SR (super resolution) image is generated by super resolution processing using a Gauss-Seidel iteration method such that a feedback value calculated by a super resolution processor 51<SB>0</SB>is added to an SR image stored in an SR image buffer 55, an SR image obtained by first addition processing is added to a feedback value calculated by the next super resolution processor 51<SB>1</SB>, and further, an SR image obtained by second addition processing is added to a feedback value calculated by the next super resolution processor 51<SB>2</SB>. The generated SR image is outputted to the outside, supplied to the SR image buffer 55 through a switch 53 and used in the next super resolution processing. The present invention is applicable to a device handling an image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program that can speed up a process for generating a high-resolution image.

  Super-resolution is a technique for generating a high-resolution image from a low-resolution image.

  Super-resolution is a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) by finding the pixel value of each pixel in a high-resolution image of one frame from multiple overlapping low-resolution images. This is a technique for reconstructing an image having a resolution higher than that of the image sensor. For example, super-resolution is used when generating a high-resolution satellite photograph.

  1 and 2 are diagrams illustrating the principle of super-resolution.

  1, a, b, c, d, e, and f shown at the top of FIG. 1 and FIG. 2 are high ones to be obtained from a low resolution image (LR (Low Resolution) image) obtained by imaging a certain subject. It represents the pixel value of a resolution image (SR (Super Resolution) image), that is, the pixel value of each pixel when the subject is pixelated with the same resolution as the resolution of the SR image.

  For example, when the width of one pixel of the image sensor is only the width of two pixels constituting the subject and the subject cannot be captured at its resolution, as shown in FIG. Among the three pixels, the pixel value of A, which is a mixture of the pixel values of a and b, is captured in the left pixel, and the pixel value of B, which is a mixture of the pixel values of c and d, is captured in the center pixel. In the right pixel, a C pixel value obtained by mixing e and f pixel values is captured. A, B, and C represent pixel values of pixels constituting an LR image obtained by imaging.

  As shown in FIG. 2, the subject at a position shifted by the width of 0.5 pixels of the pixels constituting the subject with reference to the subject position in FIG. If the pixel value is shifted (taken while being shifted), the left pixel of the three pixels of the image sensor takes in the D pixel value obtained by mixing the pixel values of the half of a, the whole of b, and the half of c. In the center pixel, the pixel value of E, which is a mixture of the pixel values of half of c, the whole of d, and half of e, is captured. Further, in the pixel on the right side, the F pixel value obtained by mixing half of e and the whole pixel value of f is captured. D, E, and F also represent pixel values of pixels constituting the LR image obtained by imaging.

  The following equation (1) is obtained from the imaging result of such an LR image. By obtaining a, b, c, d, e, and f from Equation (1), it is possible to obtain an image with a resolution higher than that of the image sensor.

  FIG. 3 is a block diagram showing a configuration example of a conventional image processing apparatus 1 that generates an SR image by super-resolution processing by back projection.

  The image processing apparatus 1 of FIG. 3 is provided in, for example, a digital camera and processes a still image obtained by imaging.

As shown in FIG. 3, the image processing apparatus 1 is composed of the super-resolution processor 11 ₀ to 11 _2, summing circuit 12, adder circuit 13 and the SR image buffer 14,.

LR ₀ which is an LR image obtained by imaging is input to the super-resolution processor 11 ₀ , and LR ₁ is input to the super-resolution processor 11 ₁ . LR ₂ is input to the super-resolution processor 11 ₂ . LR _{0 to} LR ₂ are images that are continuously captured, and each has an overlapping imaging range. When images are taken continuously, the range of the subject appearing in the image of the imaged result is usually slightly shifted due to camera shake or the like, and does not completely match but partially overlaps.

Super-resolution processor 11 _0, the LR _0, based on the SR image stored in the SR image buffer 14, generates a difference image representing a difference between them, and outputs the feedback value to the summing circuit 12. The feedback value is a value representing a difference image having the same resolution as the SR image.

The SR image buffer 14 stores an SR image generated by the super-resolution process performed immediately before. Immediately after the start of processing, when no SR image has been generated yet, for example, an image obtained by upsampling LR ₀ to an image having the same resolution as the SR image is stored in the SR image buffer 14. The

Similarly, the super-resolution processor 11 ₁ generates a difference image representing the difference between LR ₁ and the SR image stored in the SR image buffer 14, and provides a feedback value representing the generated difference image. The result is output to the summing circuit 12.

The super-resolution processor 11 ₂ generates a difference image representing the difference between the LR ₂ and the SR image stored in the SR image buffer 14, and adds a feedback value representing the generated difference image to the summing circuit 12. Output to.

Summing circuit 12 averages the supplied feedback value from the super-resolution processor 11 ₀ to 11 _2, obtained by averaging, and outputs the image of the same resolution as that of the SR image in the adder circuit 13.

  The adder circuit 13 adds the SR image stored in the SR image buffer 14 and the SR image supplied from the summing circuit 12, and outputs the SR image obtained by the addition. The output of the adding circuit 13 is supplied to the outside of the image processing apparatus 1 as a result of the super-resolution processing, and is also supplied to and stored in the SR image buffer 14.

FIG. 4 is a block diagram showing a configuration example of the super-resolution processor 11 _n (super-resolution processors 11 ₀ , 11 ₁ , 11 ₂ ).

As shown in FIG. 4, the super-resolution processor 11 _n includes a motion vector detection circuit 21, a motion compensation circuit 22, a downsampling filter 23, an addition circuit 24, an upsampling filter 25, and a backward motion compensation circuit 26. Is done.

The SR image read from the SR image buffer 14 is input to the motion vector detection circuit 21 and the motion compensation circuit 22, and LR _n obtained by imaging is input to the motion vector detection circuit 21 and the addition circuit 24.

The motion vector detection circuit 21 detects a motion vector based on the SR image based on the input SR image and LR _n , and outputs the detected motion vector to the motion compensation circuit 22 and the backward motion compensation circuit 26. .

The motion compensation circuit 22 performs motion compensation on the SR image based on the motion vector supplied from the motion vector detection circuit 21, and outputs an image obtained by performing motion compensation to the downsampling filter 23. The position of the object in the image obtained by performing motion compensation is close to the position of the object in LR _n .

Downsampling filter 23 generates an image at the same resolution as LR _n by downsampling the image supplied from the motion compensation circuit 22, and outputs the generated image to the adder circuit 24. Obtaining a motion vector from the SR image and LR _n and making an image obtained by performing motion compensation using the obtained motion vector an image having the same resolution as the LR image means that the image obtained by imaging is stored in the SR image buffer 14. This corresponds to simulation based on the stored SR image.

The adder circuit 24 generates a difference image representing the difference between LR _n and the image simulated in this way, and outputs the generated difference image to the upsampling filter 25.

  The upsampling filter 25 generates an image having the same resolution as the SR image by upsampling the difference image supplied from the adder circuit 24, and outputs the generated image to the backward motion compensation circuit 26.

  The reverse motion compensation circuit 26 performs reverse motion compensation on the image supplied from the upsampling filter 25 based on the motion vector supplied from the motion vector detection circuit 21, and obtains the reverse motion compensation. A feedback value representing the received image is output to the summing circuit 12. The position of the object shown in the image obtained by performing the motion compensation in the reverse direction is close to the position of the object shown in the SR image stored in the SR image buffer 14.

“Improving Resolution by Image Registration”, MICHAL IRANI AND SHMUEL PELEG, Department of Computer Science, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel, Communicated by Rama Chellapa, Received June 16, 1989; accepted May 25, 1990

The super-resolution processing by back projection is usually repeated a plurality of times to obtain one frame of SR image having sufficient resolution. For example, if the SR image generated by the image processing apparatus 1 of FIG. 3 and output from the adder circuit 13 does not have sufficient resolution, the SR image is stored in the SR image buffer 14 and then the super solution. re-used as the input image processing unit 11 ₀ to 11 _2, super-resolution process is repeated.

  Therefore, it takes time to obtain an SR image having a sufficient resolution.

  The present invention has been made in view of such a situation, and makes it possible to speed up a process of generating a high-resolution image.

  The image processing apparatus according to one aspect of the present invention provides a pixel of the second resolution difference image that represents a difference between the input first resolution image and a second resolution image higher than the first resolution. Are added as pixels of the input image of the second resolution, and a plurality of addition means for performing the addition process, each of the different first resolution images and the second image obtained by the immediately preceding addition process are included. Image processing means is provided for generating the second resolution image as a processing result by performing the addition processing for the second and subsequent times with the resolution image as an input, and performing the addition processing a predetermined number of times.

  Control means for controlling, based on the image of the second resolution obtained by the addition process, whether or not to perform the next addition process by using the image of the second resolution obtained by the addition process as an input. Can be further provided.

  Adjustment means for adjusting the gain of the signal representing the difference image can be further provided.

  At least one of adjustment of gain of a signal representing the second resolution image input as an image used to obtain the difference image and adjustment of gain of a signal representing the obtained difference image Further adjustment means can be provided.

  Generation means for generating an initial image of the second resolution that becomes an input of the first addition process can be further provided. When the image processing means generates the second resolution image of the processing result of the nth frame using the captured image of the first resolution of the nth frame, the generation means includes n− Some pixels constituting the second resolution image of the processing result of the first frame are replaced with pixels constituting the image obtained by upsampling the image of the first resolution of the n frame. An image can be generated as the initial image.

  The generation means includes an upsampling processing means for upsampling the image of the first resolution of the nth frame obtained by imaging, and an image of the second resolution of the processing result of the (n-1) th frame. Correcting means for performing motion compensation on the image of the second resolution of the processing result of the (n-1) th frame using a motion vector detected based on the image obtained by upsampling by the upsampling processing means; Of the image obtained by performing motion compensation by the correcting means, the upsampling processing means for the area in the area where the object whose position has been moved by motion compensation is displayed is the corresponding area. An image that generates the initial image by replacing with pixels of the image obtained by upsampling by It may further include a forming means.

  An image processing method or program according to an aspect of the present invention performs an addition process for the second and subsequent times by inputting an image having a different first resolution and an image having a second resolution obtained by the immediately preceding addition process, A step of generating a second resolution image as a processing result by performing the addition processing a predetermined number of times;

  In one aspect of the present invention, the second and subsequent addition processes are performed by inputting an image with a different first resolution and an image with a second resolution obtained by the immediately preceding addition process, and a predetermined number of times. By performing the addition process, a second resolution image as a result of the process is generated.

  According to one aspect of the present invention, it is possible to speed up the process of generating a high-resolution image.

It is a figure which shows the principle of super-resolution. It is another figure which shows the principle of super-resolution. It is a block diagram which shows the structural example of the conventional image processing apparatus. It is a block diagram which shows the structural example of a super-resolution processor. It is a figure which shows the imaging device which concerns on one Embodiment of this invention. It is a figure which shows the example of a super-resolution process. It is a block diagram which shows the structural example of an imaging device. It is a block diagram which shows the structural example of an image process part. It is a block diagram which shows the structural example of a super-resolution processor. It is a flowchart explaining the super-resolution process of an imaging device. It is a flowchart explaining the feedback value calculation process performed in step S2 of FIG. It is a block diagram which shows the other structural example of an image process part. It is a flowchart explaining the other super-resolution process of an imaging device. It is a block diagram which shows the other structural example of a super-resolution processor. It is a figure which shows the example of the characteristic of a gain adjustment. It is a flowchart explaining another feedback value calculation process. It is a block diagram which shows the other structural example of a super-resolution processor. It is a block diagram which shows the other structural example of a super-resolution processor. It is a figure which shows the example of GUI. It is a figure which shows the example of the characteristic of a filter. It is a flowchart explaining other feedback value calculation processing. It is a block diagram which shows the other structural example of a super-resolution processor. It is a figure which shows the concept of imaging of a moving image. It is a figure which shows the concept of the imaging of the conventional moving image. It is a block diagram which shows the other structural example of an image process part. It is a block diagram which shows the structural example of an initial stage image generation circuit. It is a figure which shows the example of the production | generation of an initial image. It is a block diagram which shows the structural example of a moving image application processing circuit. It is a figure which shows the example of the production | generation of a mask image. It is a flowchart explaining the super-resolution process at the time of video imaging. FIG. 31 is a flowchart for describing initial image generation processing performed in step S101 of FIG. 30. FIG. It is a flowchart explaining the moving image application process performed in step S103 of FIG. It is a block diagram which shows the other structural example of a moving image application processing circuit. It is a figure which shows the other example of the production | generation of a mask image. It is a flowchart explaining another moving image application process. It is a block diagram which shows the other structural example of an image process part. It is a figure which shows the example of a super-resolution process. It is a figure which shows the other example of a super-resolution process. It is a block diagram which shows the other structural example of an image process part. It is a figure which shows the further another example of a super-resolution process. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

  FIG. 5 is a diagram showing an imaging device 31 according to an embodiment of the present invention.

  The imaging device 31 is a digital camera and has a function of capturing a still image in accordance with a user operation and a function of capturing a moving image at a predetermined frame rate. The imaging device 31 also has a function of generating a high-resolution SR image by super-resolution processing using the LR image obtained by such an imaging function.

  FIG. 6 is a diagram illustrating an example of super-resolution processing.

In the example of FIG. 6, image reconstruction is performed based on LR _{0 to} LR ₂ which are LR images obtained by imaging, and one frame of SR image is generated. LR _{0 to} LR ₂ are LR images captured continuously, and have overlapping imaging ranges. The vertical lines and horizontal lines that are superimposed on LR _{0 to} LR ₂ in FIG. 6 are given for convenience of explanation.

  In the imaging device 31, when an SR image is generated, super-resolution processing is performed using the SR image in addition to the LR image as shown in FIG.

  FIG. 7 is a block diagram illustrating a configuration example of the imaging device 31.

  As illustrated in FIG. 7, the imaging device 31 includes an imaging unit 41, an image processing unit 42, and a recording unit 43.

  The imaging unit 41 captures a still image and a moving image, and outputs an LR image obtained by imaging to the image processing unit 42.

  The image processing unit 42 performs super-resolution processing based on the LR image supplied from the imaging unit 41, and outputs the SR image obtained by the super-resolution processing to the recording unit 43. The super-resolution processing by the image processing unit 42 is repeated until an SR image having a target resolution is generated according to control by the recording unit 43.

  When the SR image supplied from the image processing unit 42 has a sufficient resolution, the recording unit 43 records the SR image on a predetermined recording medium such as a flash memory and controls the image processing unit 42. To stop repeating the super-resolution process. The resolution of the SR image recorded on the recording medium is higher than the resolution of the image sensor provided in the imaging unit 41.

  FIG. 8 is a block diagram illustrating a configuration example of the image processing unit 42 of FIG.

The configuration of the image processing unit 42 illustrated in FIG. 8 is prepared, for example, as a function for capturing a still image, and includes super-resolution processors 51 _{0 to} 51 ₂ , addition circuits 52 _{0 to} 52 ₂ , a switch 53, and the like. An initial image generation circuit 54 and an SR image buffer 55 are included.

In the example of FIG. 8, the feedback value obtained by the super-resolution processor 51 ₀ is added to the SR image stored in the SR image buffer 55, the SR image obtained by the first addition process following Is added to the feedback value obtained by the super-resolution processor 51 ₁ , and the SR image obtained by the second addition processing is added to the feedback value obtained by the next super-resolution processor 51 _2. As described above, an SR image is generated by super-resolution processing using the Gauss-Seidel method.

  In the conventional configuration shown in FIG. 3, a plurality of feedback values obtained based on the LR image and the SR image are averaged in the summing circuit 12 and then added to the SR image. The feedback value obtained by the next super-resolution processor is immediately added to the SR image represented by the feedback value obtained based on the LR image and the SR image. As a result, the number of feedbacks is increased, and the time until the SR image having a sufficient resolution can be obtained (time until convergence) can be shortened. That is, the processing for obtaining the SR image can be speeded up.

Hereinafter, as appropriate, using the input LR image and the SR image is called a feedback value arithmetic processing process for obtaining the feedback values to be performed by the super-resolution processor 51 ₀ to 51 _2, an image represented by the feedback value SR The process of generating an SR image of one frame by adding to an image is called an addition process. The super-resolution process is realized by repeating the feedback value calculation process and the addition process.

LR ₀ that is an LR image obtained by the imaging unit 41 is input to the super-resolution processor 51 ₀ , and LR ₁ is input to the super-resolution processor 51 ₁ . LR ₂ is input to the super-resolution processor 51 ₂ . LR _{0 to} LR ₂ are LR images taken continuously, and each has an overlapping imaging range.

The super-resolution processor 51 ₀ performs feedback value calculation processing based on LR ₀ and the SR image stored in the SR image buffer 55, and adds a feedback value representing an image having the same resolution as the SR image to the adder circuit 52 _0. Output to.

Addition circuit 52 ₀ is stored in adds the SR image represented by the feedback value supplied from the SR image and super resolution processor 51 ₀ stored in the SR image buffer 55 (SR image buffer 55 Add the pixel value of the pixel included in the SR image represented by the feedback value as the pixel value of the pixel that is not in the SR image), and output the 1-frame SR image obtained by the addition as the result of the first addition process To do. The SR image outputted from the addition circuit 52 ₀ is input to the addition circuit 52 ₁ and the super-resolution processor 51 _1.

Super-resolution processor 51 _1, the LR _1, performs the feedback value arithmetic processing based on the SR image supplied from the addition circuit 52 _0, and outputs a feedback value to the addition circuit 52 _1.

The adder circuit 52 ₁ adds the SR image supplied from the adder circuit 52 ₀ and the SR image represented by the feedback value supplied from the super-resolution processor 51 ₁ , and adds one frame of SR obtained by the addition. The image is output as a result of the second addition process. The SR image output from the adder circuit 52 _{1 is} input to the super-resolution processor 51 ₂ and the adder circuit 52 ₂ .

Super-resolution processor 51 _2, and LR _2, performs the feedback value arithmetic processing based on the SR image supplied from the addition circuit 52 _1, and outputs a feedback value to the addition circuit 52 _2.

The adder circuit 52 ₂ adds the SR image supplied from the adder circuit 52 ₁ and the SR image represented by the feedback value supplied from the super-resolution processor 51 ₂ , and adds the SR image obtained by the addition to 3 The result of the addition process, that is, the result of the super-resolution process is output. The SR image outputted from the addition circuit 52 ₂ is inputted to the recording unit 43, is supplied to the SR image buffer 55 through the switch 53, it is stored.

When the initial image is generated by the initial image generation circuit 54, the switch 53 is connected to the terminal a and stores the initial image in the SR image buffer 55. The switch 53, when the SR image obtained as a result of the super-resolution processing is supplied from the addition circuit 52 ₂ is connected to the terminal b, and stores the SR image in the SR image buffer 55.

The initial image generation circuit 54 generates an initial image by, for example, upsampling LR ₀ to an image having the same resolution as that of the SR image when starting to generate an SR image of one frame by super-resolution processing. The generated initial image is stored in the SR image buffer 55 via the switch 53 connected to the terminal a.

SR image buffer 55, the initial image generated by the initial image generating circuit 54 or stores the SR image supplied from the addition circuit 52 _2.

FIG. 9 is a block diagram illustrating a configuration example of the super-resolution processor 51 _n (super-resolution processors 51 ₀ , 51 ₁ , 51 ₂ ).

As shown in FIG. 9, the super-resolution processor 51 _n has the same configuration as that of FIG. 4, and includes a motion vector detection circuit 61, a motion compensation circuit 62, a downsampling filter 63, an adder circuit 64, An upsampling filter 65 and a backward motion compensation circuit 66 are included.

The SR image supplied for use in the calculation of the feedback value is input to the motion vector detection circuit 61 and the motion compensation circuit 62, and LR _n obtained by imaging is input to the motion vector detection circuit 61 and the addition circuit 64. . The motion vector detection circuit 61 and the motion compensation circuit 62 of the super-resolution processor 51 ₁ is input SR image stored in the SR image buffer 55, the super-resolution processor 51 ₁ or the super-resolution processor 51 ₂ The motion vector detection circuit 61 and the motion compensation circuit 62 receive the SR images output from the addition circuits 52 ₀ and 52 ₁ provided in the preceding stage.

The motion vector detection circuit 61 detects a motion vector based on the SR image based on the input SR image and LR _n , and outputs the detected motion vector to the motion compensation circuit 62 and the backward motion compensation circuit 66. .

The motion compensation circuit 62 performs motion compensation on the SR image based on the motion vector supplied from the motion vector detection circuit 61, and outputs an image obtained by performing motion compensation to the downsampling filter 63. The position of the object in the image obtained by performing motion compensation is close to the position of the object in LR _n .

Downsampling filter 63 generates an image at the same resolution as LR _n by downsampling the image supplied from the motion compensation circuit 62, and outputs the generated image to the addition circuit 64. Obtaining a motion vector from the SR image and LR _n and making the image obtained by motion compensation using the obtained motion vector the same resolution as the LR image is to simulate the image obtained by capturing based on the SR image. Is equivalent to

The adder circuit 64 generates a difference image representing a difference between LR _n and the image simulated in this manner, and outputs the generated difference image to the upsampling filter 65.

  The upsampling filter 65 generates an image having the same resolution as the SR image by upsampling the difference image supplied from the adder circuit 64 and outputs the generated image to the backward motion compensation circuit 66.

The reverse motion compensation circuit 66 performs reverse motion compensation on the image supplied from the upsampling filter 65 based on the motion vector supplied from the motion vector detection circuit 61, and obtains the reverse motion compensation. A feedback value representing the received image is output to the adding circuit 52 _n . The position of the object shown in the image obtained by performing the motion compensation in the reverse direction is close to the position of the object shown in the SR image.

  Here, the super-resolution processing performed by the image processing unit 42 having the configuration of FIG. 8 will be described with reference to the flowchart of FIG.

In this process, a still image is picked up by the image pickup unit 41, LR ₀ is the super-resolution processor 51 ₀ , LR ₁ is the super-resolution processor 51 ₁ , and LR ₂ is the super-resolution processor 51 _2. It is started when each is input.

In step S1, the initial image generation circuit 54 generates an initial image, for example, by upsampling LR ₀ to an image having the same resolution as the SR image, and the generated initial image is input to the SR image buffer 55 via the switch 53. Remember. LR ₀ obtained by the imaging unit 41 is also input to the initial image generation circuit 54.

In step S2, the super-resolution processor 51 ₀ performs a feedback value calculation process based on LR ₀ and the SR image stored in the SR image buffer 55, and outputs the feedback value to the adding circuit 52 ₀ . The feedback value calculation process will be described later with reference to the flowchart of FIG.

In step S3, the addition circuit 52 _0, and the SR image stored in the SR image buffer 55, determined by carried feedback value calculation processing in step S2, supplied from the super resolution processor 51 ₀ feedback value The SR images represented by the above are added, and one frame of the SR image obtained by the addition is output as a result of the first addition process.

In step S4, the super-resolution processor 51 _1, the LR _1, it performs the feedback value arithmetic processing based on the SR image supplied from the addition circuit 52 _0, and outputs a feedback value to the addition circuit 52 _1.

In step S5, the addition circuit 52 _1, and the SR image supplied from the addition circuit 52 _0, determined by the feedback value calculation process performed in step S4, the feedback value supplied from the super resolution processor 51 ₁ The represented SR images are added, and the one-frame SR image obtained by the addition is output as a result of the second addition process.

In step S6, the super-resolution processor 51 _2, and LR _2, performs the feedback value arithmetic processing based on the SR image supplied from the addition circuit 52 _1, and outputs a feedback value to the addition circuit 52 _2.

In step S7, the addition circuit 52 _2, and the SR image supplied from the addition circuit 52 _1, determined by the feedback value calculation process performed in step S6, the feedback value supplied from the super resolution processor 51 ₂ Add the represented SR images.

In step S8, the addition circuit 52 _2, and the SR image supplied from the addition circuit 52 _1, SR image obtained by adding the SR image represented by the feedback value supplied from the super-resolution processor 51 ₂ Is output to the recording unit 43 as a result of the super-resolution processing and stored in the SR image buffer 55.

Thereafter, the recording unit 43, when the SR image outputted from the addition circuit 52 ₂ is whether it has sufficient resolution has been determined, it is determined not to have a sufficient resolution, the process steps Returning to S2, the above process is repeated. In the first feedback value calculation processing and addition processing of the super-resolution processing that is repeatedly performed, the SR image stored in the SR image buffer 55 in step S8 is used.

On the other hand, if it is determined that the SR image outputted from the addition circuit 52 ₂ has a sufficient resolution, to repeat the super-resolution processing is stopped, the image having the sufficient resolution are recorded.

  Next, feedback value calculation processing performed in steps S2, S4, and S6 of FIG. 10 will be described with reference to the flowchart of FIG.

In step S11, the motion vector detection circuit 61 detects a motion vector based on the SR image based on the input SR image and LR _n , and uses the detected motion vector as the motion compensation circuit 62 and the backward motion compensation circuit. 66.

  In step S 12, the motion compensation circuit 62 performs motion compensation on the SR image based on the motion vector supplied from the motion vector detection circuit 61, and outputs an image obtained by performing motion compensation to the downsampling filter 63.

In step S _< b _> 13, the downsampling filter 63 generates an image having the same resolution as LRn by downsampling the image supplied from the motion compensation circuit 62, and outputs the generated image to the addition circuit 64.

In step S _< b _> 14, the adder circuit 64 generates a difference image representing a difference between the input LR _n and the image supplied from the downsampling filter 63 as a downsampling result, and outputs the generated difference image to the upsampling filter 65. To do.

  In step S <b> 15, the upsampling filter 65 generates an image having the same resolution as the SR image by upsampling the difference image supplied from the addition circuit 64, and outputs the generated image to the backward motion compensation circuit 66.

In step S <b> 16, the backward motion compensation circuit 66 performs backward motion compensation on the image supplied from the upsampling filter 65 as an upsampling result based on the motion vector supplied from the motion vector detection circuit 61. A feedback value representing an image obtained by performing the direction motion compensation is output to the adder circuit 52 _n . Thereafter, the process returns to steps S2, S4, and S6 of FIG. 10, and the subsequent processing is performed.

  FIG. 12 is a block diagram illustrating another configuration example of the image processing unit 42. The same components as those in FIG. 8 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

Configuration of the image processing unit 42 in FIG. 12, the loop control circuit 71 ₀ and 71 _1, switch 72 ₀ and 72 ₁ is further provided, the switch by the switching and loop control circuit 71 ₁ of the switch 72 ₀ by loop control circuit 71 ₀ 8 is different from the configuration of FIG. 8 in that whether the result of the addition process is output to the recording unit 43 or used for the next addition process is controlled according to the switching of 72 ₁ .

That is, the image processing unit 42 in FIG. 12, according to the switching of the switch 72 ₀ by loop control circuit 71 _0, the first addition processing result outputted from the addition circuit 52 ₀ is used for the second addition processing Whether the signal is output to the recording unit 43 as it is or not, and the result of the second addition process output from the adder circuit 52 ₁ is 3 in accordance with the switching of the switch 72 ₁ by the loop control circuit 71 _1. Whether it is used for the addition process for the second time or whether it is output to the recording unit 43 as it is is controlled.

For example, the first addition result as sufficient resolution by the second and subsequent feedback value arithmetic processing when the SR image is not obtained in the process, it performs addition processing, second addition output from the addition circuit 52 ₁ Only when the SR image having sufficient resolution is not obtained as a result of the processing, the processing using the LR images of LR _{0 to} LR ₂ is completed so that the third feedback value calculation processing and addition processing are performed. Check the resolution of the SR image every time, and if sufficient resolution is not obtained, perform the following feedback value calculation processing and addition processing, and conversely, if sufficient resolution is obtained, perform processing there. By terminating the processing, the feedback value calculation process and the addition process can be completed with a small number of times, and the super-resolution process can be speeded up.

The super-resolution processor 51 ₀ performs a feedback value calculation process based on LR ₀ and the SR image stored in the SR image buffer 55 and outputs the feedback value to the adder circuit 52 ₀ .

Addition circuit 52 ₀ adds the SR image represented by the feedback value supplied from the SR image and super resolution processor 51 ₀ stored in the SR image buffer 55, one frame obtained by the addition The SR image is output as a result of the first addition process. The SR image outputted from the addition circuit 52 ₀ is supplied to the loop control circuit 71 ₀ and the switch 72 _0.

Loop control circuit 71 _0, when check the resolution (image quality) of the SR image supplied from the addition circuit 52 _0, and sufficient resolution can be obtained, by connecting the switch 72 ₀ to the terminal a, The SR image as a result of the first addition process is output to the recording unit 43. Further, the loop control circuit 71 _0, if sufficient resolution is not obtained, the switch 72 ₀ by connecting the terminal b, and first results of the SR image addition processing super-resolution processor 51 ₁ and the adder circuit 52 ₁ output.

For loop control circuit 71 _0, for example, it is given a reference of the SR image used to check the resolution of the SR image, the difference between the SR image of the reference, the minimum square error, etc. covariance Based on this, the output destination of the SR image as a result of the addition process is determined. For example, when the difference from the reference SR image is smaller than the difference set as the threshold value, the SR image as a result of the addition process is output to the recording unit 43. For example, as the reference of the SR image, or an SR image recorded in the SR image buffer 55, SR image outputted from one of the super-resolution processor 52 ₀ to 52 ₂ is to be selected as appropriate May be.

Super-resolution processor 51 _1, the LR _1, performs the feedback value arithmetic processing based on the SR image supplied from the addition circuit 52 ₀ through a switch 72 _0, and outputs a feedback value to the addition circuit 52 _1.

The adder circuit 52 ₁ adds the SR image supplied from the adder circuit 52 ₀ via the switch 72 ₀ and the SR image represented by the feedback value supplied from the super-resolution processor 51 ₁ and adds them. The obtained one-frame SR image is output as a result of the second addition process. The SR image output from the adder circuit 52 ₁ is supplied to the loop control circuit 71 ₁ and the switch 72 ₁ .

The loop control circuit 71 ₁ confirms the resolution of the SR image supplied from the adder circuit 52 ₁ , and if a sufficient resolution is obtained, the switch 72 ₁ is connected to the terminal a to connect the second time. The SR image as a result of the addition process is output to the recording unit 43. Further, when a sufficient resolution is not obtained, the loop control circuit 71 ₁ connects the switch 72 ₁ to the terminal b to thereby superimpose the SR image as a result of the second addition process on the super-resolution processor 51. ₂ and the adder circuit 52 ₂ .

Also for the loop control circuit 71 _1, SR image of the reference used to check the resolution of the SR image is given.

The super-resolution processor 51 ₂ performs a feedback value calculation process based on the LR ₂ and the SR image supplied from the adder circuit 52 ₁ via the switch 72 ₁ , and outputs the feedback value to the adder circuit 52 ₂ .

The adder circuit 52 ₂ adds the SR image supplied from the adder circuit 52 ₁ via the switch 72 ₁ and the SR image represented by the feedback value supplied from the super-resolution processor 51 ₂ and adds them. The obtained one-frame SR image is output as a result of the super-resolution processing. The SR image outputted from the addition circuit 52 ₂ is supplied to the recording unit 43, is supplied to the SR image buffer 55 through the switch 53, it is stored.

Incidentally, the super-resolution processor 51 ₀ to 51 ₂ in FIG. 12 also each have the same configuration as the configuration shown in FIG.

  Here, the super-resolution processing performed by the image processing unit 42 having the configuration of FIG. 12 will be described with reference to the flowchart of FIG.

Also in this processing, a still image is picked up by the image pickup unit 41, LR ₀ is a super-resolution processor 51 ₀ , LR ₁ is a super-resolution processor 51 ₁ , and LR ₂ is a super-resolution processor 51 ₂ . Started when entered.

In step S 21, the initial image generation circuit 54 generates an initial image by upsampling LR _{0 or} the like, and stores the generated initial image in the SR image buffer 55 via the switch 53.

In step S22, the super-resolution processor 51 _0, the LR _0, it performs the feedback value arithmetic processing based on the SR image stored in the SR image buffer 55, and outputs a feedback value to the addition circuit 52 _0. Here, the same processing as that described with reference to FIG. 11 is performed.

In step S23, the adder circuit 52 ₀ is determined by the feedback value calculation process performed in step S22, stores the SR image represented by the feedback value supplied from the super-resolution processor 51 ₀ in the SR image buffer 55 The one-frame SR image obtained by the addition is output as a result of the first addition process.

In step S24, the loop control circuit 71 ₀ checks the image quality of the SR image supplied from the addition circuit 52 _0.

In step S25, the loop control circuit 71 ₀ determines based on whether repeat such feedback value arithmetic processing check result of the image quality of the SR image.

If it is determined that repetition of the feedback value calculation process in step S25, in step S26, the super-resolution processor 51 _1, the LR _1, based on the SR image supplied from the addition circuit 52 ₀ through the switch 72 ₀ It performs the feedback value arithmetic processing Te, outputs the feedback value to the addition circuit 52 _1.

In step S27, the adding circuit 52 ₁ obtains the SR image obtained by the feedback value calculation process performed in step S26 and represented by the feedback value supplied from the super-resolution processor 51 ₁ from the adding circuit 52 _0. It adds to the supplied SR image, and outputs the SR image obtained by the addition as a result of the second addition process.

In step S28, the loop control circuit 71 ₁ checks the image quality of the SR image supplied from the addition circuit 52 ₁ .

In step S29, the loop control circuit 71 ₁ determines based on whether repeat such feedback value arithmetic processing check result of the image quality of the SR image.

If it is determined in step S29 that the feedback value calculation process is to be repeated, in step S30, the super-resolution processor 51 ₂ is based on LR ₂ and the SR image supplied from the adder circuit 52 ₁ via the switch 72 _1. It performs the feedback value arithmetic processing Te, outputs the feedback value to the addition circuit 52 _2.

In step S31, the addition circuit 52 ₂ is determined by the feedback value calculation process performed in step S30, the SR image represented by the feedback value supplied from the super-resolution processor 51 _2, from the addition circuit 52 ₁ Add to the supplied SR image.

In step S32, the addition circuit 52 _2, and the SR image supplied from the addition circuit 52 _1, SR image obtained by adding the SR image represented by the feedback value supplied from the super-resolution processor 51 ₂ Is output to the recording unit 43 as a result of the super-resolution processing and stored in the SR image buffer 55. Thereafter, the process returns to step S22, and the above processes are repeated.

  On the other hand, when it is determined in steps S25 and S29 that the feedback value calculation process or the like is not repeated, the SR image determined to have sufficient image quality is output to the recording unit 43, and the process ends.

FIG. 14 is a block diagram showing another configuration example of the super-resolution processor 51 _n . The same components as those in FIG. 9 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

The super-resolution processor 51 _n shown in FIG. 14 is different from the configuration shown in FIG. 9 in that a feedback gain control circuit 81 is further provided at the subsequent stage of the backward motion compensation circuit 66.

Feedback gain control circuit 81 adjusts the gain of the feedback value supplied from the backward motion compensation circuit 66, and outputs a feedback value obtained by adjusting the gain to the addition circuit 52 _n. From the backward motion compensation circuit 66, an image obtained by performing backward motion compensation on an image obtained by up-sampling the difference image between the LR _n and SR images is supplied as a feedback value.

  For example, the feedback gain control circuit 81 adjusts the gain by making the overall frequency component of the image signal of the SR image represented by the feedback value supplied from the backward motion compensation circuit 66 an arbitrary multiple, The gain is adjusted by multiplying a different value for each frequency component.

  Also, the feedback gain control circuit 81 lowers the gain of a signal having a frequency component that is likely to adversely affect the image quality when the gain is increased and added to the SR image. Whether or not the image quality is adversely affected is determined by, for example, the amount of noise included in the feedback value supplied from the backward motion compensation circuit 66, the SAD (Sum of Absolute Difference) required when detecting the motion vector, or the like. Judgment is made based on the reliability of the motion vector. Note that the process of adjusting the gain may be performed uniformly on the entire image or may be performed for each region.

  15A to 15D are diagrams illustrating examples of gain adjustment characteristics realized by the feedback gain control circuit 81. FIG.

  The horizontal axis represents frequency, and the vertical axis represents gain. The characteristics shown in FIG. 15A lower the gain of the low frequency component and increase the gain of the high frequency component. The characteristics shown in FIG. 15B increase the gain of the low frequency component and decrease the gain of the high frequency component. The characteristics shown in FIG. 15C increase the gain of the frequency component near the middle and decrease the gain of the other components. The characteristic shown in FIG. 15D lowers the gain of all frequency components.

Here, a feedback value calculation process performed by the super-resolution processor 51 _n having the configuration of FIG. 14 will be described with reference to the flowchart of FIG. This process is also performed in steps S2, S4, and S6 of FIG.

The process shown in FIG. 16 is the same as the process described with reference to FIG. 11 except that a process for adjusting the gain is added. That is, in step S41, the motion vector detecting circuit 61 detects a motion vector based on the inputted SR image and LR _n, and outputs the detected motion vector to the motion compensation circuit 62 in the backward motion compensation circuit 66.

  In step S 42, the motion compensation circuit 62 performs motion compensation on the SR image based on the motion vector supplied from the motion vector detection circuit 61, and outputs an image obtained by performing motion compensation to the downsampling filter 63.

In step S43, the down-sampling filter 63 generates an image at the same resolution as LR _n by downsampling the image supplied from the motion compensation circuit 62, and outputs the generated image to the addition circuit 64.

In step S44, the adder circuit 64 generates a difference image representing a difference between the input LR _n and the image supplied from the downsampling filter 63 as a downsampling result, and outputs the generated difference image to the upsampling filter 65. To do.

  In step S <b> 45, the upsampling filter 65 generates an image having the same resolution as the SR image by upsampling the image supplied from the addition circuit 64, and outputs the generated image to the backward motion compensation circuit 66.

  In step S 46, the backward motion compensation circuit 66 performs backward motion compensation on the image supplied from the upsampling filter 65 as an upsampling result based on the motion vector supplied from the motion vector detection circuit 61. An image obtained by performing the direction motion compensation is output to the feedback gain control circuit 81.

In step S47, the feedback gain control circuit 81 adjusts the gain of the signal representing the image as the result of the backward motion compensation supplied from the backward motion compensation circuit 66, and the feedback obtained by adjusting the gain. The value is output to the adder circuit 52 _n . Thereafter, the process returns to steps S2, S4, and S6 of FIG. 10, and the subsequent processing is performed.

FIG. 17 is a block diagram showing still another configuration example of the super-resolution processor 51 _n . The same components as those in FIG. 14 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

The super-resolution processor 51 _n shown in FIG. 17 is different from the configuration shown in FIG. 14 in that the feedback gain control circuit 81 is provided not between the backward motion compensation circuit 66 but between the adder circuit 64 and the upsampling filter 65. And different. That is, in the configuration of FIG. 17, the difference image obtained by the addition circuit 64 is supplied to the feedback gain control circuit 81, and gain adjustment is performed on the difference image at this stage.

  As described above, the insertion position of the feedback gain control circuit 81 is not limited to the position following the backward motion compensation circuit 66. When the feedback gain control circuit 81 is inserted between the adder circuit 64 and the upsampling filter 65 as shown in FIG. 17, the function of the upsampling filter 65 and the function of the feedback gain control circuit 81 are realized by one circuit. You may do it.

FIG. 18 is a block diagram showing another configuration example of the super-resolution processor 51 _n . The same components as those in FIG. 9 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

The super-resolution processor 51 _n shown in FIG. 18 is different from that shown in FIG. 9 in that a taste control circuit 101 is provided before the motion compensation circuit 62 and a taste control circuit 102 is provided after the backward motion compensation circuit 66. Different from the configuration.

  As shown in FIG. 18, the SR image used for the calculation of the feedback value is supplied to the motion vector detection circuit 61 and the taste control circuit 101. The taste control circuit 101 is also supplied with taste parameters in accordance with user operations. The taste parameter is also supplied to the taste control circuit 102.

The taste parameter represents, for example, a value adjusted by the user using a GUI (Graphical User Interface) as shown in FIG. 19, and a filter used for processing is selected and selected according to this value. An SR image having an image quality corresponding to the filtered filter is generated by the super-resolution processor 51 _n . In the example of FIG. 19, the user can adjust the value by sliding the slide bar 111, and the current adjustment value is displayed on the display unit 112.

  Returning to the description of FIG. 18, the taste control circuit 101 selects a filter according to a taste parameter, and filters the SR image using the selected filter. The taste control circuit 101 outputs the SR image obtained by performing filtering to the motion compensation circuit 62.

In the subsequent circuit, processing similar to that described above is performed. That is, in the motion compensation circuit 62, motion compensation is performed on the SR image supplied from the taste control circuit 101, and an image obtained by performing motion compensation is down-sampled by the down-sampling filter 63. Difference image representing a difference between LR _n in the adder circuit 64 is generated, with respect to up-sampled image a difference image in the up-sampling filter 65, the backward motion compensation is performed in the backward motion compensation circuit 66 . The SR image obtained by performing the motion compensation in the reverse direction is supplied to the taste control circuit 102.

The taste control circuit 102 selects a filter according to the taste parameter, and performs filtering on the SR image supplied from the backward motion compensation circuit 66 using the selected filter. The taste control circuit 102 outputs the SR image obtained by filtering to the adder circuit 52 _n as a feedback value.

  20A to 20D are diagrams illustrating examples of characteristics of filters selected in the taste control circuits 101 and 102. FIG.

  The characteristics shown in FIGS. 20A to 20D are the same as the characteristics shown in FIGS. 15A to 15D, respectively, and the characteristics of such a filter are adjusted according to taste parameters as shown in FIG. 20A.

For example, when the image quality of the SR image output from the adder circuit 52 _n has a lot of glare, and the user adjusts to reduce it, the taste control circuit 102 performs LPF (Low Pass Filter). Is selected, and the image quality of the SR image output from the adding circuit 52 _n is adjusted to be milder.

Here, with reference to the flowchart of FIG. 21, the feedback value calculation process performed by the super-resolution processor 51 _n having the configuration of FIG. 18 will be described.

  The process shown in FIG. 21 is the same as the process described with reference to FIG. 11 and the like except that a process for adjusting the image quality is added.

  In step S51, the taste control circuit 101 selects a filter according to the taste parameter, and filters the supplied SR image using the selected filter (performs taste control). The taste control circuit 101 outputs the SR image obtained by performing filtering to the motion compensation circuit 62.

In step S52, the motion vector detecting circuit 61 detects a motion vector based on the inputted SR image and LR _n, and outputs the detected motion vector to the motion compensation circuit 62 in the backward motion compensation circuit 66.

  In step S53, the motion compensation circuit 62 performs taste control based on the motion vector supplied from the motion vector detection circuit 61, performs motion compensation on the SR image supplied from the taste control circuit 101, and performs motion compensation. The obtained image is output to the downsampling filter 63.

In step S _< b _> 54, the downsampling filter 63 generates an image having the same resolution as LRn by downsampling the image supplied from the motion compensation circuit 62, and outputs the generated image to the addition circuit 64.

In step S55, the adder circuit 64 generates a difference image representing a difference between the input LR _n and the image supplied from the downsampling filter 63 as a downsampling result, and outputs the generated difference image to the upsampling filter 65. To do.

  In step S <b> 56, the upsampling filter 65 generates an image having the same resolution as the SR image by upsampling the image supplied from the addition circuit 64, and outputs the generated image to the backward motion compensation circuit 66.

  In step S57, the backward motion compensation circuit 66 performs motion compensation in the backward direction on the image supplied from the upsampling filter 65 as an upsampling result based on the motion vector supplied from the motion vector detection circuit 61. An image obtained by performing the direction motion compensation is output to the taste control circuit 102.

In step S58, the taste control circuit 102 selects a filter according to the taste parameter, and performs filtering on the SR image supplied from the backward motion compensation circuit 66 using the selected filter. The taste control circuit 102 outputs the SR image obtained by filtering to the adder circuit 52 _n as a feedback value. Thereafter, the process returns to steps S2, S4, and S6 of FIG. 10, and the subsequent processing is performed.

FIG. 22 is a block diagram showing still another configuration example of the super-resolution processor 51 _n . The same components as those in FIG. 18 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

The configuration of the super-resolution processor 51 _n in FIG. 22 is that the taste control circuit 101 is provided between the down-sampling filter 63 and the addition circuit 64 instead of the previous stage of the motion compensation circuit 62, and the taste control circuit 102 is 18 is different from the configuration of FIG. 18 in that it is provided between the adder circuit 64 and the upsampling filter 65 instead of following the backward motion compensation circuit 66.

That is, in the configuration of FIG. 22, the image obtained by down-sampling filter 63 is supplied to the taste control circuit 101 are adapted to filtering is subjected to the same image resolution and LR _n. Further, the difference image obtained by the addition circuit 64 is supplied to the taste control circuit 102, and the difference image is subjected to filtering at this stage.

As described above, the taste control circuit 101 may be provided at any position in the super-resolution processor 51 _{n as} long as it is located at a stage preceding the adder circuit 64 that is configured to obtain a difference image. Further, the taste control circuit 102 may be provided at any position in the super-resolution processor 51 _{n as} long as it is a position subsequent to the adder circuit 64. For example, the taste control circuits 101 and 102 can be realized by a synthesis filter with DSF and USF.

  Note that only one of the taste control circuit 101 and the taste control circuit 102 may be provided.

  Next, a configuration for realizing a moving image imaging function will be described.

  FIG. 23 is a diagram illustrating the concept of moving image capturing performed by the image capturing apparatus 31.

LR _{n to} LR _{n + 5 in} FIG. 23 are LR images picked up by the image pickup unit 41, and are used for super-resolution processing. In the imaging unit 41, for example, LR images are captured at a frame rate of 30 frames / second.

As shown in FIG. 23, in each super-resolution process, except for the first process, three frames of LR images and one frame of SR image that are continuously captured are used. For example, in the first super-resolution processing, three frames of LR images of LR _n , LR _{n + 1} , and LR _{n + 2} are used, and SR _n that is an SR image of one frame is generated.

Further, in the second super-resolution processing, three frames of LR images of LR _{n + 1} , LR _{n + 2} and LR _{n + 3} and SR images generated by the first super-resolution processing are SR. _n is used, and SR _{n + 1} that is an SR image of one frame is generated.

In the third super-resolution processing, three frames of LR images of LR _{n + 2} , LR _{n + 3} and LR _{n + 4} and SR _{n +} which is an SR image generated by the second super-resolution processing are used. ₁ is used, and SR _{n + 2} that is an SR image of one frame is generated.

  As described above, in the imaging device 31, the SR image generated by the super-resolution processing is also used for the super-resolution processing immediately after, thereby, for example, using only the LR image as shown in FIG. Compared with the case where super-resolution processing is performed, the processing can be speeded up.

FIG. 24 is a diagram showing a concept of a conventional moving image capturing function using super-resolution processing. In each super-resolution processing, only three frames of LR images captured in succession are used. Yes. For example, in the first super-resolution process, three frames of LR images LR _n , LR _{n + 1} , and LR _{n + 2} are used, and SR _n that is a one-frame SR image is generated. In the second super-resolution processing, three frames of LR images of LR _{n + 1} , LR _{n + 2} , and LR _{n + 3} are used, and SR _{n + 1} that is an SR image of one frame is generated. ing.

  25, as shown in FIG. 23, super-resolution processing is performed using three frames of LR images captured continuously and one frame of SR image obtained by the previous super-resolution processing, It is a block diagram which shows the structural example of the image process part 42 which implement | achieves imaging of a moving image.

  The image processing unit 42 in FIG. 25 is basically an extension of the configuration in FIG. 3 that realizes still image capturing to realize moving image capturing.

As shown in FIG. 25, the image processing unit 42 includes super-resolution processors 201 _{0 to} 201 ₂ , moving image application processing circuits 202 ₁ and 202 ₂ , a summing circuit 203, an adding circuit 204, a switch 205, an initial image generating circuit. 206 and SR image buffer 207.

LR _n which is an LR image obtained by imaging is input to the super-resolution processor 201 ₀ , the moving image application processing circuits 202 ₁ and 202 ₂ , and the initial image generation circuit 206, and LR _{n + 1} is the super-resolution processor. 2011 ₁ and the moving image application processing circuit 202 ₁ . LR _{n + 2} is input to the super-resolution processor 201 ₂ and the moving image application processing circuit 202 ₂ .

Super-resolution processor 201 _0, and LR _n, it performs the feedback value arithmetic processing based on the SR image stored in the SR image buffer 207, and outputs the feedback value to the summing circuit 203. No significant difference in the content that is reflected in the LR _n and SR image input to the super-resolution processor 201 _0, SR image represented by the feedback value obtained by the super-resolution processor 201 ₀ Basically the information of all the pixels from could be used in addition processing, the output of the super-resolution processor 201 ₀ is different from the output of the super-resolution processor 201 _1, 201 _2, the moving picture application processing Without being applied, it is output to the summing circuit 203 as it is.

The super-resolution processor 201 ₁ performs feedback value calculation processing based on LR _{n + 1} and the SR image stored in the SR image buffer 207, and outputs a feedback value representing the SR image to the moving image application processing circuit 202 ₁ . To do.

The super-resolution processor 201 ₂ performs feedback value calculation processing based on LR _{n + 2} and the SR image stored in the SR image buffer 207, and outputs a feedback value representing the SR image to the moving image application processing circuit 202 ₂ . To do.

The moving image application processing circuit 202 ₁ determines that it can be used to add to the SR image stored in the SR image buffer 207 out of the SR image pixels supplied from the super-resolution processor 2011 ₁ Is output to the summing circuit 203 as a feedback value.

  For example, a pixel in a region that is considered to deteriorate the image quality when added to the SR image stored in the SR image buffer 207 is determined as a pixel that cannot be used for addition to the SR image. Only pixels in a region that is thought to contribute to improving the image quality are extracted by the moving image application process. In moving image capturing, a subject with a large amount of motion often becomes a subject, and when motion compensation is performed as one process in the feedback value calculation processing, an image in which the subject is blurred can be obtained. Therefore, this moving image application process removes pixels in such a blurred area. Details of the moving image application processing will be described later.

The moving image application processing circuit 202 ₂ determines that it can be used to add to the SR image stored in the SR image buffer 207 among the pixels of the SR image supplied from the super-resolution processor 201 _2. Is output to the summing circuit 203 as a feedback value.

The summing circuit 203 averages the feedback values supplied from the super-resolution processor 201 ₀ and the moving image application processing circuits 202 ₁ and 202 ₂ , and adds an image having the same resolution as the SR image obtained by averaging. Output to.

  The adder circuit 204 adds the SR image stored in the SR image buffer 207 and the SR image supplied from the summing circuit 203, and outputs the SR image obtained by the addition. The output of the adder circuit 204 is supplied to the recording unit 43 as a result of the super-resolution processing, and also supplied to the SR image buffer 207 and stored therein.

  When the initial image is generated by the initial image generation circuit 206, the switch 205 is connected to the terminal a and stores the initial image in the SR image buffer 207. Further, when the SR image is supplied from the addition circuit 204, the switch 205 is connected to the terminal b and stores the SR image in the SR image buffer 207. Of the repeated super-resolution processing, the switch 205 is connected to the terminal a when the first super-resolution processing is performed, and is connected to the terminal b when the second and subsequent super-resolution processing is performed.

The initial image generation circuit 206 generates and generates an initial image based on LR _n obtained by imaging and SR _n-1 which is the SR image of the immediately preceding frame obtained as a result of the super-resolution processing. The initial image is stored in the SR image buffer 207 via the switch 205 connected to the terminal a. For example, when the SR image of the second frame is to be generated by super-resolution processing, LR ₂ obtained by imaging and SR ₁ which is the SR image obtained as a result of the super-resolution processing are the initial images. When the SR image of the third frame supplied to the generation circuit 206 is to be generated by super-resolution processing, the LR ₃ obtained by imaging and the SR image obtained as a result of the super-resolution processing SR ₂ is supplied.

  The SR image buffer 207 stores the initial image generated by the initial image generation circuit 206 or the SR image output from the addition circuit 204.

  FIG. 26 is a block diagram illustrating a configuration example of the initial image generation circuit 206.

As shown in FIG. 26, the initial image generation circuit 206 includes an upsampling processing unit 221, a motion correction unit 222, and an image generation unit 223. LR _n is input to the upsampling processing unit 221, and SR _n−1 is input to the motion correction unit 222.

The upsampling processing unit 221 upsamples LR _n into an image having the same resolution as the SR image, and outputs the image obtained by the upsampling to the motion correction unit 222 and the image generation unit 223.

Motion motion compensation unit 222, the SR _n-1, supplied from the up-sampling processing section 221, on the basis of the up-sampling result of LR _n, detects a motion vector based on the up-sampling result of LR _n, detected Motion compensation is performed on SR _n-1 using a vector. The motion correction unit 222 outputs an image obtained by performing motion compensation to the image generation unit 223.

The image generation unit 223 generates an initial image based on the upsampling result of LR _n supplied from the upsampling processing unit 221 and the result of motion compensation supplied from the motion correction unit 222, and the generated initial image is SR Store in the image buffer 207. For example, the image generation unit 223 uses the SR image of the LR _n upsampling result of the region at the same position as the pixel in the region where the motion compensation cannot be performed well among the SR images supplied from the motion correction unit 222. An initial image is generated by substituting with the pixels.

  27A to 27C are diagrams illustrating an example of initial image generation.

LR _n shown on the left side of FIG. 27A is an LR image input to the upsampling processing unit 221, and SR _n−1 shown on the right side is an SR image input to the motion correction unit 222. A difference in size between LR _n and SR _n-1 represents a difference in resolution. Since LR _n is taken after LR _n-1 used to generate SR _n-1 , the position of the car in LR _n moves in the direction of travel from the position of the car in SR _n-1. It is in the position. The background of LR _n and SR _n-1 is almost the same.

For example, such LR _n is upsampled motion vector is detected by using the up-sampling result and SR _n-1. The detected motion vector is a vector heading from the position of the car shown in SR _n-1 to the position of the car shown in the upsampling result of LR _n . When motion compensation is performed on SR _n−1 using such a motion vector, an SR image as shown in FIG. 27B is obtained as a result of motion compensation.

The SR image shown in FIG. 27B is obtained by moving the position of the automobile in the traveling direction as compared with SR _{n-1 in} FIG. 27A. The region A ₁ is a background region that is not shown in SR _n−1 , and this region A ₁ is extracted as a region where motion compensation cannot be performed well.

The SR image shown in FIG. 27C is an image in which the pixels in the region A _{1 in} the SR image shown in FIG. 27B are replaced with the pixels in the region at the same position as the region A _{1 in the} upsampling result of LR _n. It is. In this way, an SR image in which pixels in a region where motion compensation cannot be performed successfully is replaced with pixels of the upsampling result of LR _n is used as an initial image. Note that the frame representing the area A ₁ shown in the initial image of FIG. 27C is attached for convenience of explanation, and is not actually shown in the image.

  Such initial image generation is performed at the start of the first super-resolution process among the repeated super-resolution processes.

Figure 28 is a block diagram showing a configuration example of a moving picture application processing circuit 202 _1.

As illustrated in FIG. 28, the moving image application processing circuit 202 ₁ includes a difference image generation unit 231, a binarization processing unit 232, an enlargement processing unit 233, and a mask processing unit 234.

LR _n and LR _{n + 1} obtained by imaging is input to the differential image generating unit 231, SR image obtained by the feedback value arithmetic processing by the super-resolution processor 201 ₁ is performed mask processing unit 234 Is input.

The difference image generation unit 231 obtains a difference image representing the difference between LR _n and LR _{n + 1} and outputs the obtained difference image to the binarization processing unit 232.

Binarization processing unit 232, based on the supplied difference image from the difference image generation unit 231, for example, the area difference between LR _n and LR _{n + 1} is almost no 1, the area difference is the threshold at 0 A binary image to be expressed is generated, and the generated binary image is output to the enlargement processing unit 233.

  The enlargement processing unit 233 enlarges the binarized image generated by the binarization processing unit 232 to an image having the same resolution as the SR image, and outputs the image to the mask processing unit 234 as a mask image.

The mask processing unit 234, by using the mask image supplied from the expansion processing unit 233 performs a masking process on the SR image supplied from the super-resolution processor 201 _1, summing the result of performing a masking process as a feedback value Output to the circuit 203.

  29A to 29C are diagrams illustrating an example of generation of a mask image.

LR _{n + 1} shown on the left side of FIG. 29A and LR _n shown on the right side are LR images input to the difference image generation unit 231. For example, a difference image is obtained from such an LR image and binarized.

FIG. 29B is a diagram illustrating an example of a binarized image. Of the entire binarized image, the region A ₁₁ corresponds to a region in which the background is reflected in both the LR _{n + 1} and LR _n LR images. This area is represented by 1 because there is almost no difference.

On the other hand, since the area A ₁₂ is an area corresponding to the area where the car is shown in LR _{n + 1 and the} area _where the background is shown in LR _n , there is a difference, and this area is represented by 0. The Since the area A ₁₃ is an area corresponding to the area where the background is shown in LR _{n + 1 and the} area _where the car is shown in LR _n , there is a difference, and this area is also represented by 0.

  The SR image shown in FIG. 29C is a mask image obtained by enlarging the binarized image shown in FIG. 29B to an image having the same resolution as the SR image. Mask processing using such a mask image is performed on the SR image obtained by the feedback value calculation processing. In the SR image obtained by the feedback value calculation process, the pixels in the area corresponding to the area represented by black in the mask image in FIG. 29C are replaced with 0 by the mask process, and a part of them is replaced with 0. The obtained SR image is output to the summing circuit 203 as a feedback value.

  By performing such a mask process on the result of the feedback value calculation process, it is possible to prevent the pixel information of the region of the subject that moves rapidly from being fed back and added to the SR image stored in the SR image buffer 207. be able to. Since the area of a subject that moves rapidly may cause blurring of the subject, it is possible to prevent the image quality of the SR image from deteriorating by preventing feedback of pixel information in such an area. it can.

Note that the moving image application processing circuit 202 ₂ has the same configuration as that of FIG. In the moving picture application processing circuit 202 _2, LR _{n + 2} is used instead of LR _{n + 1,} the same processing as the above-described process is performed.

  Here, with reference to the flowchart of FIG. 30, the super-resolution processing at the time of moving image capturing performed by the image processing unit 42 having the configuration of FIG. 25 will be described.

In this process, the moving image is picked up by the image pickup unit 41, and LR _n is input to the super-resolution processor 201 ₀ , the moving image application processing circuits 202 ₁ and 202 ₂ , and the initial image generation circuit 206, and LR _{n + 1.} Is input to the super-resolution processor 201 ₁ and the moving image application processing circuit 202 ₁ , and LR _{n + 2} is input to the super-resolution processor 201 ₂ and the moving image application processing circuit 202 ₂ .

  In step S <b> 101, the initial image generation circuit 206 executes an initial image generation process and stores the generated initial image in the SR image buffer 207. The initial image generation process will be described later with reference to the flowchart of FIG.

In step S102, the super-resolution processor 201 ₀ to 201 _2, respectively, it performs the feedback value arithmetic processing based on the SR image stored in the LR image and the SR image buffer 207, and outputs a feedback value.

Based on LR _n and the SR image in the super-resolution processor 201 _0, based on LR _{n + 1} and the SR image in the super-resolution processor 201 _1, the super-resolution processor 201 ₂ LR _{n +} Based on ₂ and the SR image, processing similar to that described with reference to FIG. 11 is performed. Feedback value outputted from the super-resolution processor 201 ₀ is input to the summing circuit 203, a feedback value outputted from the super-resolution processor 201 ₁ is input to the moving picture application processing circuit 202 _1. Feedback value outputted from the super-resolution processor 201 ₂ is inputted to the moving picture application processing circuit 202 _2.

In step S103, the moving image application processing circuits 202 ₁ and 202 ₂ each perform moving image application processing, and output the feedback value obtained by the moving image application processing to the summing circuit 203. The moving image application process will be described later with reference to the flowchart of FIG.

In step S < _b > 104, the summing circuit 203 sums the feedback values supplied from the super-resolution processor 201 ₀ and the moving image application processing circuits 202 ₁ and 202 ₂ , and outputs the summed result to the adding circuit 204.

  In step S <b> 105, the adding circuit 204 adds the SR image represented by the feedback value supplied from the summing circuit 203 to the SR image stored in the SR image buffer 207.

  In step S106, the addition circuit 204 outputs the SR image obtained by the addition to the recording unit 43, and supplies the SR image to the SR image buffer 207 for storage.

  Until it is determined that the resolution of the SR image output from the image processing unit 42 has a sufficient resolution, the processes in and after step S102 are repeated.

  Next, the initial image generation process performed in step S101 in FIG. 30 will be described with reference to the flowchart in FIG.

In step S111, the up-sampling processing unit 221 of the initial image generating circuit 206, LR _n and up-sampled to the same resolution of the SR image, the motion compensation unit 222 and the image generating unit an image obtained by upsampling To 223.

In step S112, the motion correction unit 222 detects a motion vector based on SR _n−1 and the upsampling result of LR _n supplied from the upsampling processing unit 221.

In step S _< b _> 113, the motion correction unit 222 performs motion compensation on SR _n−1 using the detected motion vector, and outputs the obtained image to the image generation unit 223.

In step S _< b _> 114, the image generation unit 223 compares the SR image of the upsampling result of LR _n supplied from the upsampling processing unit 221 with the SR image of the result of motion compensation supplied from the motion correction unit 222. A region where motion compensation cannot be performed is extracted from the SR image as a result of compensation.

In step S115, the image generation unit 223, a pixel of the extracted area in step S114, the area at the same position, to generate the initial image by replacing a pixel of LR _n upsampling result of the SR image. Thereafter, the process returns to step S101 in FIG. 30, and the subsequent processing is performed.

Next, the moving image application process performed in step S103 in FIG. 30 will be described with reference to the flowchart in FIG. Here, a description will be given of a process moving picture application processing circuit 202 ₁ performs. Similar processing is performed in the moving image application processing circuit 202 ₂ .

In step S121, the differential image generator 231 of the moving picture application processing circuit 202 ₁ calculates a difference image representing a difference between LR _n and LR _{n + 1,} and outputs a difference image obtained in the binarization processing unit 232.

In step S122, the binarization processing unit 232, based on the difference image supplied from the difference image generation unit 231, for example, the area difference between LR _n and LR _{n + 1} is almost no 1, the difference is the threshold A binarized image in which the area is represented by 0 is generated, and the generated binarized image is output to the enlargement processing unit 233.

  In step S123, the enlargement processing unit 233 enlarges the binarized image generated by the binarization processing unit 232 to an image having the same resolution (size) as the SR image, and outputs the image as a mask image to the mask processing unit 234.

In step S124, the mask processing unit 234, by using the mask image supplied from the expansion processing unit 233 performs a masking process on the SR image supplied from the super-resolution processor 201 _1, a result of performing the masking process A feedback value is output to the summing circuit 203.

Figure 33 is a block diagram showing another configuration example of the moving picture application processing circuit 202 _1.

In the example of FIG. 25, LR _n is used for the moving image application processing circuits 202 ₁ and 202 ₂ , LR _{n + 1} is used for the moving image application processing circuit 202 ₁ , and LR _{n + 2} is the moving image as LR images used for the moving image application processing. It was assumed to be input to the application processing circuit 202 _2, but instead of LR _n, it is also possible to have SR image stored in the SR image buffer 207 is input.

In the example of FIG. 33, the moving image application processing circuit 202 ₁ includes an upsampling processing unit 241, a difference image generation unit 242, a binarization processing unit 243, and a mask processing unit 244, and is obtained by imaging. The LR _{n + 1} is input to the upsampling processing unit 241, and the SR image stored in the SR image buffer 207 is input to the difference image generation unit 242. SR image obtained by the feedback value arithmetic processing by the super-resolution processor 201 ₁ is performed is input to the mask processing unit 244.

The upsampling processing unit 241 upsamples LR _{n + 1} into an image having the same resolution as that of the SR image, and outputs an image obtained by the upsampling to the difference image generation unit 242.

The difference image generation unit 242 obtains a difference image representing the difference between the LR _{n + 1} upsampling result and the SR image, and outputs the obtained difference image to the binarization processing unit 243.

Based on the difference image supplied from the difference image generation unit 242, for example, the binarization processing unit 243 has an area where there is almost no difference between the upsampling result of LR _{n + 1} and the SR image, and the difference is less than the threshold value. A binarized image representing a region with 0 is generated, and the generated binarized image is output to the mask processing unit 244 as a mask image.

The mask processing unit 244, by using the mask image supplied from the binarization processing unit 243 performs a masking process to the SR image supplied from the super-resolution processor 201 _1, a result of performing the masking process A feedback value is output to the summing circuit 203.

  34A to 34C are diagrams illustrating examples of mask image generation.

LR _{n + 1} shown on the left side of FIG. 34A is an LR image input to the upsampling processing unit 241, and the SR image shown on the right side is an SR image input to the difference image generation unit 242. For example, such an LR image is upsampled, a difference image is obtained from the upsampling result and the SR image, and binarized.

FIG. 34B is a diagram illustrating an example of a binarized image. Of the entire binarized image, the region A ₂₁ has a background in both the LR _{n + 1} upsampling result and the SR image. Since this is an area corresponding to the area and there is almost no difference, this area is represented by 1.

On the other hand, since the region A ₂₂ is a region corresponding to the region where the car is shown in the upsampling result of LR _{n + 1 and} the region where the background is shown in the SR image, there is a difference. It is represented by 0. The region A ₂₃ is a region corresponding to the region where the background is reflected in the upsampling result of LR _{n + 1} , and the region corresponding to the region where the car is reflected in the SR image, so there is a difference, and this region is also 0. expressed.

The SR image shown in FIG. 34C is a mask image obtained from the binarized image shown in FIG. 34B. Against SR image obtained by the super-resolution processor 201 _1, a mask process using such a mask image is performed. In the SR image obtained by the feedback value calculation process, the pixels in the area corresponding to the area represented by black in the mask image in FIG. 34C are replaced with 0 by the mask process, and a part of them is replaced with 0. The obtained SR image is output to the summing circuit 203 as a feedback value.

  By performing such a mask process on the result of the feedback value calculation process, it is possible to prevent the information on the pixels in the region of the subject that moves rapidly from being used as the feedback value.

The moving picture application processing circuit 202 ₂ can also have the same configuration as that of FIG. In this case, in the moving picture application processing circuit 202 _2, LR _{n + 2} is used instead of LR _{n + 1,} the same processing as the above-described process is performed.

Next, the moving image application processing performed by the moving image application processing circuit 202 ₁ having the configuration of FIG. 33 will be described with reference to the flowchart of FIG.

This process is a process performed in step S103 of FIG. Here, a description will be given of a process moving picture application processing circuit 202 ₁ performs. Similar processing is performed in the moving image application processing circuit 202 ₂ .

In step S131, the upsampling processing unit 241 upsamples LR _{n + 1} into an image having the same resolution as the SR image, and outputs the image obtained by the upsampling to the difference image generation unit 242.

In step S132, the difference image generation unit 242 obtains a difference image representing the difference between the LR _{n + 1} upsampling result and the SR image, and outputs the obtained difference image to the binarization processing unit 243.

  In step S133, the binarization processing unit 243 generates a mask image by binarizing the difference image supplied from the difference image generation unit 242, and outputs the generated mask image to the mask processing unit 244.

In step S134, the mask processing unit 244, by using the mask image supplied from the binarization processing unit 243 performs a masking process on the SR image supplied from the super-resolution processor 201 _1, was subjected to mask processing The result is output to the summing circuit 203 as a feedback value.

  FIG. 36 is a block diagram illustrating another configuration example of the image processing unit 42 that realizes capturing of a moving image.

  In the image processing unit 42 in FIG. 36, each addition process is performed using three frames of LR images captured continuously and one frame of SR image obtained by the immediately preceding addition process. The same components as those in FIG. 25 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

In the example of FIG. 36, the image processing unit 42 includes super-resolution processors 201 _{0 to} 201 ₂ , moving image application processing circuits 202 ₁ and 202 ₂ , addition circuits 251 _{0 to} 251 ₂ , a switch 205, and an initial image generation circuit 206. , And SR image buffer 207.

  An image processing unit 42 in FIG. 36 is an extension of the configuration in FIG. 8 that generates SR images by super-resolution processing using the Gauss-Seidel method as an implementation of moving image capturing.

LR _n that is an LR image supplied from the imaging unit 41 is input to the super-resolution processor 201 ₀ , the moving image application processing circuits 202 ₁ and 202 ₂ , and the initial image generation circuit 206, and LR _{n + 1} is super-resolution. The data is input to the processor 201 ₁ and the moving image application processing circuit 202 ₁ . LR _{n + 2} is input to the super-resolution processor 201 ₂ and the moving image application processing circuit 202 ₂ . LR _{n to} LR _{n + 2} are LR images captured continuously.

The super-resolution processor 201 ₀ performs a feedback value calculation process based on LR _n and the SR image stored in the SR image buffer 207, and outputs the feedback value to the adder circuit 251 ₀ .

The addition circuit 251 ₀ adds the SR image stored in the SR image buffer 207 and the SR image represented by the feedback value supplied from the super-resolution processor 201 ₀ , and adds one frame obtained by the addition. The SR image is output as a result of the first addition process. The SR image output from the adder circuit 251 _{0 is} input to the super-resolution processor 201 ₁ and the adder circuit 251 ₁ .

The super-resolution processor 201 ₁ performs feedback value calculation processing based on LR _{n + 1} and the SR image supplied from the addition circuit 251 _0, and outputs the feedback value to the moving image application processing circuit 202 ₁ .

The moving image application processing circuit 202 ₁ uses only the information of the pixels determined to be usable for addition to the SR image among the pixels of the SR image supplied from the super-resolution processor 2011 ₁ as a feedback value. 251 output to ₁ .

The adder circuit 251 ₁ adds the SR image supplied from the adder circuit 251 ₀ and the SR image represented by the feedback value supplied from the moving image application processing circuit 202 ₁ , and adds one frame of the SR image obtained by the addition. Is output as a result of the second addition process. The SR image output from the adder circuit 251 _{1 is} input to the super-resolution processor 201 ₂ and the adder circuit 251 ₂ .

The super-resolution processor 201 ₂ performs a feedback value calculation process based on LR _{n + 2} and the SR image supplied from the adder circuit 251 _1, and outputs the feedback value to the moving image application processing circuit 202 ₂ .

The adding circuit 251 ₂ adds the SR image supplied from the adding circuit 251 ₁ and the SR image represented by the feedback value supplied from the moving image application processing circuit 202 ₂ , and adds one frame of the SR image obtained by the addition. Is output as a result of the super-resolution processing. The SR image outputted from the addition circuit 251 ₂ is inputted to the recording unit 43, is supplied to the SR image buffer 207 through the switch 205, it is stored.

When the initial image is generated by the initial image generation circuit 206, the switch 205 is connected to the terminal a and stores the initial image in the SR image buffer 207. The switch 205, when the SR image supplied from the addition circuit 251 ₂ is connected to the terminal b, and stores the SR image in the SR image buffer 207.

  When starting the super-resolution processing, the initial image generation circuit 206 generates an initial image, for example, as described with reference to FIG. 31, and switches the switch 205 that connects the generated initial image to the terminal a. And stored in the SR image buffer 207.

SR image buffer 207, an initial image generated by the initial image generating circuit 206, or to store the SR image supplied from the addition circuit 251 _2.

  In the example of FIG. 36, one series of SR images and three frames of LR images are used to perform a series of super-resolution processing. However, the number of LR images used for the super-resolution processing is arbitrary. It is.

  For example, as shown in FIG. 37, each super-resolution process is performed by using one frame of SR image and two frames of LR image generated by the previous super-resolution process. As shown in FIG. 38, it is also possible to use one frame of SR image and one frame of LR image generated by the previous super-resolution processing. It is.

In the example of FIG. 37, each super-resolution process is performed by using two frames of LR images and one frame of SR images that are continuously captured, except for the first process. . In the first super-resolution processing, two frames of LR images, LR _n and LR _{n + 1} , are used, and SR _n that is one frame of SR image is generated.

In the second super-resolution process, two frames of LR images LR _{n + 1} and LR _{n + 2} and SR _n which is an SR image obtained by the first super-resolution process are used. SR _{n + 1} that is an SR image of one frame is generated.

In the third super-resolution process, two frames of LR images LR _{n + 2} and LR _{n + 3} and SR _{n + 1} which is an SR image obtained by the second super-resolution process are used. SR _{n + 2} that is an SR image of one frame is generated.

In the example of FIG. 38, each super-resolution process is performed by using one frame of the LR image and one frame of the SR image except for the first process. In the first super-resolution processing, one frame LR image of LR _n is used, and SR _n that is one frame SR image is generated. For example, SR _n is generated by up-sampling LR _n to an image having the same resolution as the SR image.

In the second super-resolution processing, one frame LR image of LR _{n + 1} and SR _n which is an SR image obtained by the first super-resolution processing are used, and one frame SR image is used. SR _{n + 1} is generated.

In the third super-resolution process, one frame LR image of LR _{n + 2} and SR _{n + 1} which is an SR image obtained by the second super-resolution process are used, and one frame SR image is used. SR _{n + 2} is generated.

  FIG. 39 shows still another configuration of the image processing unit 42 that performs super-resolution processing using one frame of LR image and one frame of SR image to realize moving image capturing, as shown in FIG. It is a block diagram which shows an example. The same components as those in FIG. 36 are denoted by the same reference numerals. The overlapping description will be omitted as appropriate.

The image processing unit 42 in FIG. 39 basically includes a super-resolution processor 201 ₁ , a moving image application processing circuit 202 ₁ , an addition circuit 251 ₁ , which are configurations for performing a second feedback value calculation process and an addition process. 36 except that the super-resolution processor 201 ₂ , the moving image application processing circuit 202 ₂ , and the addition circuit 251 ₂ , which are configurations for performing the third feedback value calculation process and addition process, are excluded. It has the same configuration as the configuration.

The super-resolution processor 201 ₀ performs a feedback value calculation process based on LR _n and the SR image stored in the SR image buffer 207, and outputs the result of the feedback value calculation process to the adder circuit 251 ₀ .

Addition circuit 251 ₀ adds the SR image represented by the feedback value supplied from the SR image and super-resolution processor 201 ₀ stored in the SR image buffer 207, an SR image obtained by the addition Output. The SR image outputted from the addition circuit 251 ₀ is supplied to the recording unit 43 as a result of super-resolution processing, is supplied to the SR image buffer 207 through the switch 205, it is stored.

  Thus, by reducing the number of LR images used for one super-resolution process, it is possible to speed up the super-resolution process.

As shown in FIG. 37, the image processing unit 42 that performs super-resolution processing using one frame of the SR image and two frames of the LR image is different from the configuration of the image processing unit 42 in FIG. The super-resolution processor 201 ₂ , the moving image application processing circuit 202 ₂ , and the addition circuit 251 ₂ , which are configurations for performing the third feedback value calculation process and addition process, are excluded.

  In the above description, the LR image and the SR image generated by the previous super-resolution process are used to perform the respective super-resolution processes. In the case of having a function capable of performing the above-described imaging, the super-resolution processing may be performed using an LR image and one frame of a still image captured at the same timing. In this case, the resolution of the still image captured during the capturing of the moving image is greater than the resolution of the LR image.

  FIG. 40 is a diagram illustrating a concept of super-resolution processing performed using an LR image and one frame of a still image captured during moving image capturing.

In the example of FIG. 40, the fourth super-resolution processing is performed at the same time as the LR _{n + 3} , LR _{n + 4} , and LR _{n + 5} three-frame LR images and LR _{n + 3.} This is performed using a still image _{n + 3} that is a still image of a frame.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 41 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program.

  A CPU (Central Processing Unit) 301 executes various processes according to a program stored in a ROM (Read Only Memory) 302 or a storage unit 308. A RAM (Random Access Memory) 303 appropriately stores programs executed by the CPU 301 and data. The CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.

  An input / output interface 305 is also connected to the CPU 301 via the bus 304. To the input / output interface 305, an input unit 306 including a keyboard, a mouse, and a microphone, and an output unit 307 including a display and a speaker are connected. The CPU 301 executes various processes in response to commands input from the input unit 306. Then, the CPU 301 outputs the processing result to the output unit 307.

  The storage unit 308 connected to the input / output interface 305 includes, for example, a hard disk, and stores programs executed by the CPU 301 and various data. The communication unit 309 communicates with an external device via a network such as the Internet or a local area network.

  A drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 308 as necessary.

  As shown in FIG. 41, a program recording medium for storing a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (Digital Versatile Disc), a magneto-optical disk, a removable medium 311 which is a package medium made of a semiconductor memory or the like, a ROM 302 in which a program is temporarily or permanently stored, or a storage unit 308 It is comprised by the hard disk etc. which comprise. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 309 that is an interface such as a router or a modem as necessary. Done.

  In this specification, the steps for describing a program are not only processes performed in time series in the order described, but also processes that are executed in parallel or individually even if they are not necessarily processed in time series. Is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

31 imaging device 41 imaging unit, 42 image processing unit, 43 recording unit, 51 ₀ to 51 ₂ super-resolution processor, 52 ₀ to 52 ₂ addition circuit, 53 a switch, 54 an initial image generation circuit, 55 SR image buffer, 61 motion vector detection circuit, 62 motion compensation circuit, 63 down sampling filter, 64 addition circuit, 65 up sampling filter, 66 reverse motion compensation circuit

Claims (8)

Pixels of the second resolution difference image representing the difference between the input first resolution image and the second resolution image higher than the first resolution are input to the second resolution image. Including a plurality of addition means for performing addition processing to be added as image pixels;
The addition process is performed for the second and subsequent times by inputting the different images of the first resolution and the image of the second resolution obtained by the immediately preceding addition process, and the addition process is performed a predetermined number of times. An image processing apparatus comprising image processing means for generating an image of the second resolution as a processing result. Control means for controlling, based on the image of the second resolution obtained by the addition process, whether or not to perform the next addition process by using the image of the second resolution obtained by the addition process as an input. The image processing apparatus according to claim 1. The image processing apparatus according to claim 1, further comprising an adjusting unit configured to adjust a gain of a signal representing the difference image. At least one of adjustment of gain of a signal representing the second resolution image input as an image used to obtain the difference image and adjustment of gain of a signal representing the obtained difference image The image processing apparatus according to claim 1, further comprising adjustment means for performing the adjustment. A generating unit configured to generate an initial image of the second resolution to be an input of the first addition process;
When the image processing means generates the second resolution image of the processing result of the nth frame, using the captured image of the first resolution of the nth frame,
The generation means is an image obtained by up-sampling a part of pixels constituting the second resolution image of the processing result of the (n-1) th frame and the first resolution image of the nth frame. The image processing apparatus according to claim 1, wherein an image replaced with pixels constituting the image is generated as the initial image. The generating means includes
Upsampling processing means for upsampling an image of the first resolution of the nth frame obtained by imaging;
The processing result of the (n-1) th frame using the image of the second resolution of the processing result of the (n-1) th frame and the motion vector detected based on the image obtained by the upsampling by the upsampling processing means Correction means for performing motion compensation on the image of the second resolution
Of the image obtained by performing motion compensation by the correcting means, the upsampling processing means for the area in the area where the object whose position has been moved by motion compensation is displayed is the corresponding area. The image processing apparatus according to claim 5, further comprising: an image generation unit configured to generate the initial image by replacing with a pixel of an image obtained by upsampling according to. Pixels of the second resolution difference image representing the difference between the input first resolution image and the second resolution image higher than the first resolution are input to the second resolution image. In an image processing method of an image processing apparatus including a plurality of addition means for performing addition processing to be added as image pixels,
The addition process is performed for the second and subsequent times by inputting the different images of the first resolution and the image of the second resolution obtained by the immediately preceding addition process, and the addition process is performed a predetermined number of times. An image processing method including a step of generating an image of the second resolution as a processing result. Pixels of the second resolution difference image representing the difference between the input first resolution image and the second resolution image higher than the first resolution are input to the second resolution image. In a program for causing a computer to perform image processing of an image processing apparatus including a plurality of addition means for performing addition processing to be added as image pixels,
The addition process is performed for the second and subsequent times by inputting the different images of the first resolution and the image of the second resolution obtained by the immediately preceding addition process, and the addition process is performed a predetermined number of times. Thereby generating a second resolution image as a result of processing.

Images