JP2008306608A - Image processing method, image processing apparatus, and electronic device with the image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method and image processing apparatus capable of acquiring a high-precision and high-resolution image by performing on a still picture and a moving picture, respectively, super-resolution processing for acquiring a high-resolution image from a low-resolution image. <P>SOLUTION: A high-resolution updating amount calculation section 443 determines a gradient based on a differential between a low-resolution estimated image estimated from a high-resolution image and a low-resolution real image given from frame memories 41, 42, thereby calculating an updating amount to be subtracted from the high-resolution image. At such a time, the number of times of operations in the high-resolution updating amount calculation section 443 based on the high-resolution image stored in a frame memory 45 is set in such a way that the number of times of moving picture photographing becomes less than the number of times of still picture photographing. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing method and an image processing apparatus for generating a high resolution image, and more particularly to an image processing method and an image processing apparatus for performing a super-resolution process for generating a high resolution image from a plurality of low resolution images. The present invention also relates to an electronic device that records or reproduces a high resolution image generated by the image processing apparatus.

  In recent years, with the development of various digital technologies, imaging devices such as digital cameras and digital videos that take digital images with solid-state imaging devices such as CCD (Charge Coupled Device) and CMOS (Complimentary Metal Oxide Semiconductor) sensors, Display devices such as liquid crystal displays and plasma televisions for displaying are becoming widespread. An image processing technique for converting the resolution of a digital image processed by such an imaging apparatus or display apparatus into a high resolution using a plurality of digital images taken at different times has been proposed.

  As this high-resolution conversion method, super-resolution processing for estimating one high-resolution image from a plurality of low-resolution images with positional deviation has been proposed (see Non-Patent Document 1). In this super-resolution processing, the generated high-resolution image is inversely transformed to estimate the original low-resolution image for constructing the high-resolution image, and then compared with the actual low-resolution image. In addition, a method called a reconstruction type that generates a high-resolution image close to an actual value has been proposed.

  In the super-resolution processing by the reconstruction type method, first, a high resolution image acquired from a plurality of low resolution images is assumed (STEP 1). Based on the assumed high resolution image, estimation is performed by performing inverse transform to each of a plurality of low resolution images for constructing the high resolution image (STEP 2). Then, after comparing the low-resolution image after the inverse conversion and the original low-resolution image (STEP 3), the comparison result shows that the pixel position between the converted low-resolution image and the original low-resolution image is the same at each pixel position. A high-resolution image is generated so that the difference in values is small (STEP 4). Then, the processing of STEP 2 to STEP 4 is repeated so that the difference in the value at each pixel position (hereinafter referred to as “pixel value”) between the converted low-resolution image and the original low-resolution image converges. Thus, the finally obtained high resolution image can be made an image close to the real image.

  As this reconstruction type method, ML (Maximum-Likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection Onto Convex Set) method, IBP (Iterative Back Projection) method, etc. are proposed. In the ML method, a pixel value of a low resolution image (hereinafter referred to as “low resolution estimated image”) estimated from a high resolution image and a pixel value of an actual low resolution image (hereinafter referred to as “low resolution actual image”) Is a method for generating a high-resolution image that minimizes the evaluation function (see Non-Patent Document 2). That is, the super-resolution processing by the ML method is a processing method based on the principle of maximum likelihood estimation.

  In the MAP method, a high-resolution image that minimizes the evaluation function is obtained by adding the probability information of the high-resolution image to the square error of the pixel values of the low-resolution estimated image and the low-resolution real image. It is the method of producing | generating (refer nonpatent literature 3, 4). That is, in the MAP method, an optimal high resolution image is acquired by estimating a high resolution image when the appearance probability in the posterior probability distribution based on the foresight information for the low resolution real image is maximized.

  Furthermore, in the POCS method, simultaneous equations relating to the pixel values of the high-resolution image and the low-resolution image are created, and the simultaneous equations are sequentially solved to obtain the optimum value of the pixel value of the high-resolution image. This is a method for generating a resolution image (see Non-Patent Document 5). The IBP method is a method for generating a high resolution from a plurality of low resolution images that have different shooting positions but have overlapping pixel positions of a subject, that is, a super-resolution processing method using an iterative backprojection method (non-resolution method). (See Patent Document 6).

As an image processing apparatus that acquires a high-resolution image using such super-resolution processing, the high-resolution image is divided into predetermined small regions, and the estimated values for all the pixels belonging to each small region are In order to increase the speed of the super-resolution processing by approximating the estimated value for the pixel serving as the representative point in (see Patent Document 1). That is, when estimating a low-resolution image from a high-resolution image, the estimated value of each pixel of the low-resolution image estimated from one small region of the high-resolution image is estimated from the representative points in the small region. It is assumed that it is constant. Thereby, it is possible to suppress the amount of calculation required for calculation processing when estimating a low-resolution image from a high-resolution image, and it is possible to increase the speed of super-resolution processing.
Japanese Patent No. 3337575 Sung, CP Min, KP Moon, GK "Super-resolution image reconstruction: a technical overview", (Signal processing Magazine, IEEE on Volume 26, Issue 3, May 2003, P.21-36) Tom, BC Katsaggelos, AK "Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images", (Image Processing, Proceedings, International Conference on Volume 2, Oct 1995, P.23-26 ) "Extraction of high-resolution frames from video sequences" by Schultz, RR Stevenson, RL (Image Processing, IEEE Transactions on Volume 5, Issue 6, June 1996, P.996-1011) Hardie RC Barnard KJ Armstrong EE "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images" (Image Processing, IEEE Transactions on Volume 6, Issue 12, Dec. 1997 P.1621-1633) Stark, H. Oskoui, P. "High resolution image recovery from image-plane arrays, using convex projections" (J. Opt. Soc. Am. A, on vol. 6. P.1715-1726, 1989) Michel, I. Peleg, S. "Improving Resolution by Image Registration" (CVGIP: Graph Models Image Process, on vol. 53. P.231-239, Mar. 1991)

  However, in the super-resolution processing described above, the above-described STEP2 to STEP4 are performed so that the difference between the low-resolution image estimated from the high-resolution image (the converted low-resolution image) and the original low-resolution image becomes small. It is necessary to repeat the process. The acquired high-resolution image can be made to be a high-precision image close to the actual image by repeating the steps 2 to 4 repeatedly. However, such super-resolution processing is intended for still image shooting and is not intended for moving image shooting in which the time required for the shooting operation is limited.

  In addition, by increasing the number of frames of the low-resolution image for estimating the high-resolution image, a high-precision high-resolution image can be generated, but depending on the number of frames of the low-resolution image for high-resolution image estimation A large number of frame memories are required. By increasing the number of frame memories in this way, the number of iterations in super-resolution processing can be reduced. However, when configured as an apparatus, the number of frame memories increases, and the apparatus becomes large.

  In Japanese Patent Laid-Open No. 2004-228561, in order to increase the speed of super-resolution processing, a low-resolution image is estimated based on representative points by segmentation. However, although the processing time required for estimation of a low resolution image is increased, iterative processing in super-resolution processing needs to be performed as in the conventional method. Therefore, the super-resolution processing according to Patent Document 1 is also intended for still image shooting, and not for moving image shooting in which the time required for the shooting operation is limited.

  In view of such a problem, the present invention can acquire a high-resolution image with high accuracy by performing super-resolution processing for acquiring a high-resolution image from a low-resolution image for each of a still image and a moving image. An object is to provide an image processing method and an image processing apparatus. In addition, the present invention provides an electronic apparatus including an image processing device that performs high-resolution image acquisition from a low-resolution image for each of a still image and a moving image and acquires a high-resolution image with high accuracy. The purpose is to provide.

  In order to achieve the above object, an image processing apparatus according to the present invention temporarily stores the low-resolution image in an image processing apparatus including a high-resolution image generation unit that generates a high-resolution image from a plurality of frames of low-resolution images. A plurality of first frame memories, and a second frame memory for temporarily storing the high resolution image generated by the high resolution image generation unit, wherein the high resolution image generation unit includes the first frame. The high-resolution image is generated based on the low-resolution images of a plurality of frames stored in the memory, and the generated high-resolution image is output to the second frame memory and then stored in the second frame memory. When the high-resolution image is used to perform an iterative calculation process for updating the high-resolution image, and the high-resolution image of a single frame is generated and output The high-resolution image generation unit generates the high-resolution image when the iterative calculation process using the second frame memory in the high-resolution image generation unit is the first number of repetitions, and the high-resolution image of a plurality of frames is generated and output continuously. The number of iterations of the iterative calculation process using the second frame memory in the unit is a second number that is less than the first number and equal to or more than zero.

  In such an image processing apparatus, a plurality of the first frame memories are connected in series, and the continuously input low resolution images are stored for each frame, and a plurality of the first resolution memories are stored in the high resolution image generation unit. The low-resolution images stored in the first frame memory are output in parallel.

  At this time, a third frame memory that is arranged at a subsequent stage of the first frame memory and is commonly used as the first and second frame memories, and the low-resolution image from the last stage of the first frame memory. And a selection unit that selects the high resolution image generated by the high resolution image generation unit and outputs it to the third frame memory, and generates and outputs the high resolution image of a single frame When the selection unit selects the high resolution image from the high resolution image generation unit and outputs the high resolution image to the third frame memory, and continuously generates and outputs the high resolution image of a plurality of frames, The selection unit may select the low resolution image from the last stage of the first frame memory and output it to the third frame memory.

  The high-resolution image generation unit generates the high-resolution image from the plurality of low-resolution real images, and the high-resolution processing unit includes the second frame memory as a part thereof, and the high-resolution processing A low-resolution estimation unit that acquires a plurality of low-resolution estimation images by estimating to be equal to each of the plurality of low-resolution actual images from the high-resolution image acquired by the unit, and obtained by the low-resolution estimation unit An update amount calculation unit that calculates an update amount of the high-resolution image generated by the high-resolution processing unit based on a difference value between the low-resolution estimated image and the low-resolution actual image. In the unit, the high-resolution image is updated by updating the high-resolution image generated in the previous process with the update amount obtained by the update-amount calculating unit.

  In such an image processing device, the update amount calculation unit obtains a difference value between each low-resolution actual image and the corresponding low-resolution estimated image, so that each pixel position of the low-resolution actual image is obtained. A low-resolution update amount calculation unit that calculates a low-resolution update amount in the image, and an image component included in each pixel value of the low-resolution actual image and the low-resolution estimated image at the same pixel position for each of the low-resolution actual images And based on the comparison result of the image components, a weighting factor calculation unit that generates a weighting factor for multiplying the low-resolution update amount, and the low-resolution update amount calculated by the low-resolution update amount calculation unit Then, after multiplying the weighting factor calculated by the weighting factor calculating unit, the number of pixels of the high-resolution image is adjusted based on the low-resolution update amount multiplied by the weighting factor. A high-resolution update amount calculation unit that calculates the high-resolution update amount, and in the high-resolution processing unit, the high-resolution update amount calculation unit applies the high-resolution image generated in the previous process. The high-resolution image may be updated by performing the update with the calculated high-resolution update amount.

  At this time, an image obtained by dividing the image component of the pixel value of the low-resolution real image by the image component of the pixel value of the low-resolution estimated image is obtained by comparing the image components in the weighting coefficient calculation unit. If the image component ratio becomes a value close to a predetermined value of 1 or more, the weight coefficient is set to a value close to 1, and if the image component ratio becomes a value farther than the predetermined value, the weight The coefficient is set to a value close to 0.

  The comparison result of the image components may be a ratio of high-frequency components included in pixel values of the low-resolution actual image and the low-resolution estimated image, or the low-resolution actual image and the low-resolution estimated image, respectively. It may be a ratio of standard deviations depending on pixel values, or may be a ratio of edge direction components included in pixel values of the low resolution actual image and the low resolution estimated image.

  The electronic apparatus according to the present invention is an electronic apparatus provided with an image signal based on an image that becomes a plurality of frames by external input or imaging, and having a high resolution function for converting an image based on the image signal into a high resolution image. As an image processing unit that realizes a high resolution function, any of the above-described image processing apparatuses is provided, and by performing high resolution imaging processing using the image based on the image signal as the low resolution image, a desired high resolution image is obtained. It is generated.

  The image processing method of the present invention includes a first step of generating a high-resolution image based on a plurality of frames of low-resolution images, and a second step of generating an update amount of the high-resolution image generated in the first step. An image processing method for updating the high-resolution image generated in the first step according to the update amount in the second step by repeatedly performing the first step and the second step. When the high-resolution image of a single frame is generated and output, the number of repetitions of the iterative calculation process in the first step and the second step is set to the first number, and the high-resolution image of a plurality of frames is continuously generated. The second iteration in which the number of iterations of the iterative calculation process in the first step and the second step is less than the first number and equal to or greater than zero. Characterized by a.

  When generating and outputting the high-resolution image of a single frame, the number of frames of the low-resolution image used for generating the high-resolution image in the first step is a first frame number, When continuously generating and outputting the high-resolution image of the frame, the second number of frames of the low-resolution image used for generating the high-resolution image in the first step is greater than the first frame number. The number of frames may be used.

  According to the present invention, when the high-resolution image of a single frame is output and when the high-resolution image of a plurality of frames is output continuously in a short time, the number of repetitions of the iterative calculation processing in the high-resolution processing is different. And As a result, it is possible to perform resolution enhancement processing for still images and resolution enhancement processing for moving images using a circuit configuration that performs common resolution enhancement processing. Therefore, a dedicated circuit configuration is not required for each of the still image and the moving image, and an increase in the size of the apparatus can be prevented.

  In the case of outputting the high resolution image of a single frame by providing a third frame memory that is commonly used as the first and second frame memories for storing the low resolution image and the high resolution image, respectively. The number of frames of the low-resolution image used for the high-resolution processing can be different between the case where a plurality of high-resolution images are continuously output in a short time.

  That is, when the resolution enhancement process is performed on the moving image, the number of frames of the low-resolution image to be used is increased, so that a decrease in accuracy due to a decrease in the number of iterations of the iterative calculation process can be compensated. In addition, when performing high resolution processing for still images, the third frame memory used as the first frame memory for storing low resolution images when performing high resolution processing for moving images is stored as high resolution images. It can be used as a second frame memory. As a result, it is not necessary to add a new frame memory, and the apparatus can be downsized.

  Embodiments of the present invention will be described with reference to the drawings. In the following description, an image pickup apparatus such as a digital camera or a digital video provided with an image processing apparatus (hereinafter referred to as “image processing unit”) that performs the image processing method according to the present invention will be described as an example. Further, as will be described later, a display device that performs digital processing of an image, such as a liquid crystal display or a plasma television, may be used as long as it has a similar image processing device.

(Configuration of imaging device)
First, the internal configuration of the imaging apparatus will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an internal configuration of the imaging apparatus.

  The imaging apparatus of FIG. 1 receives a solid-state imaging device (image sensor) 1 such as a CCD or CMOS sensor that converts light incident from a subject into an electrical signal, and an image signal that is an analog signal output from the image sensor 1. AFE (Analog FrontEnd) 2 for converting to a digital signal, a microphone 3 for converting sound input from the outside into an electric signal, and various image processing including super-resolution processing for an image signal to be a digital signal from the AFE 2 The image processing unit 4 that performs the processing, the audio processing unit 5 that converts the audio signal that is an analog signal from the microphone 3 into a digital signal, the image signal from the image processing unit 4, and the audio signal from the audio processing unit 5 The compression processing unit 6 that performs compression encoding processing such as the MPEG (Moving Picture Experts Group) compression encoding method, and the compression encoded signal that has been compression encoded by the compression processing unit 6 The driver unit 7 to be recorded in the memory 15, the decompression processing unit 8 that decompresses and decodes the compressed encoded signal read from the external memory 20 by the driver unit 7, and the image signal obtained by decoding by the decompression processing unit 8 A display unit 9 for displaying an image, an audio output circuit unit 10 that converts an audio signal from the expansion processing unit 8 into an analog signal, and a speaker unit that reproduces and outputs audio based on the audio signal from the audio output circuit unit 10 11, a timing generator 12 that outputs a timing control signal for matching the operation timing of each block, a CPU (Central Processing Unit) 13 that controls the drive operation of the entire imaging apparatus, and each program for each operation A memory 14 for temporarily storing data during program execution, an operation unit 15 for inputting an instruction from a user, Comprises a PU13 a bus line 16 for exchanging data with each block, a memory 14 and a bus line 17 for exchanging data with each block.

(1) Moving Image Shooting Operation In this imaging apparatus, when the operation unit 15 instructs to perform a moving image shooting operation, an image signal that is an analog signal obtained by the photoelectric conversion operation of the image sensor 1 is output to the AFE 2. . At this time, the image sensor 1 is supplied with a timing control signal from the timing generator 12 to perform horizontal scanning and vertical scanning, and output an image signal as data for each pixel. In the AFE 2, when an image signal that is an analog signal is converted into a digital signal and input to the image processing unit 4, various image processing such as signal conversion processing for generating a luminance signal and a color difference signal is performed. .

  Further, in the image processing unit 4, when the operation unit 15 is required to increase the resolution of the image signal obtained from the image sensor 1 by digital zoom, super-resolution based on the image signals for a plurality of frames from the image sensor 1. Processing is performed. Further, a luminance signal and a color difference signal are generated based on the image signal that has been subjected to the super-resolution processing. Further, since the super-resolution processing is performed, as described later, the motion amount of the image signal of a plurality of frames is calculated, and the alignment of each frame is performed according to the motion amount.

  Then, the image signal subjected to the image processing by the image processing unit 4 is given to the compression processing unit 6. At this time, an audio signal which is an analog signal obtained by inputting the sound into the microphone 3 is converted into a digital signal by the audio processing unit 5 and given to the compression processing unit 6. As a result, the compression processing unit 6 compresses and encodes the image signal and the audio signal, which are digital signals, based on the MPEG compression encoding method, gives the image signal and the audio signal to the driver unit 7, and records them in the external memory 20. At this time, the compressed signal recorded in the external memory 20 is read out by the driver unit 7 and applied to the expansion processing unit 8 to be subjected to expansion processing to obtain an image signal. This image signal is given to the display unit 9 to display a subject image that is currently photographed through the image sensor 1.

(2) Still Image Shooting Operation Even when a still image shooting is instructed, unlike a case where a moving image shooting is instructed, there is no acquisition of an audio signal by the microphone 3, and a compression signal of only an image signal is received in the compression processing unit 6. It is obtained by a compression code method such as a JPEG (Joint Photographic Experts Group) compression code method, and is only recorded in the external memory 20, and its basic operation is the same as that during moving image shooting. Further, in the case of this still image shooting, not only the compressed signal for the still image shot by the operation unit 15 is recorded in the external memory 20, but also the compressed signal for the current image shot by the image sensor 1 is also stored in the external memory. 20 is temporarily recorded. As a result, the compression signal for the currently captured image is expanded by the expansion processing unit 8 so that the current image captured by the image sensor 1 is displayed on the display unit 9 and can be confirmed by the user. .

  As described above, the timing generator 12 performs timing control signals to the AFE 2, the video processing unit 4, the audio processing unit 5, the compression processing unit 6, and the decompression processing unit 8 when performing an imaging operation for each of a still image and a moving image. Is performed, and an operation synchronized with the imaging operation for each frame by the image sensor 1 is performed. Further, at the time of still image shooting, a timing control signal is given from the timing generator 12 to the image sensor 1, the AFE 2, the video processing unit 4, and the compression processing unit 6 based on the shutter operation by the operation unit 15. The operation timing of each unit is synchronized.

(3) Reproduction of moving image or still image When reproduction of a moving image or still image recorded in the external memory 20 is instructed through the operation unit 15, the compressed signal recorded in the external memory 20 is 7 and read to the decompression processing unit 8. In the case of a moving image, the decompression processing unit 8 decompresses and decodes the image based on the MPEG compression encoding method to obtain an image signal and an audio signal. Then, the image signal is given to the display unit 9 to reproduce the image, and the audio signal is given to the speaker unit 11 via the audio output circuit unit 10 to reproduce the audio. Thereby, the moving image based on the compressed signal recorded in the external memory 20 is reproduced together with the sound.

  In the case of a still image, the decompression processing unit 8 decompresses and decodes the image signal based on the JPEG compression code method, and acquires an image signal. Then, the image signal is given to the display unit 9 to reproduce the image. Thereby, a still image based on the compressed signal recorded in the external memory 20 is reproduced.

(Basic concept of super-resolution processing)
Next, the basic concept of super-resolution processing executed in the image processing unit 4 in the above-described imaging apparatus will be briefly described. In describing the basic concept of super-resolution processing, in order to simplify the description, image data is based on pixel values of a plurality of pixels lined up in a one-dimensional direction, and 2 taken at different times. It is assumed that super-resolution processing is performed using frame image data. The pixel value in the following description indicates a luminance value. FIG. 2A shows the luminance distribution of the subject imaged by the image sensor 1, and FIGS. 2B to 2D show images when the subject is imaged by the image sensor 1 at different times. Data (low-resolution actual image) is shown. FIG. 3 is a flowchart for generating image data that becomes a high-resolution image from the low-resolution real image in FIG. Further, FIG. 3 schematically shows signal transition in each flow.

  For the subject with the luminance distribution shown in FIG. 2A, the sample points of the first frame when imaged by the image sensor 1 at time T1 are S1, S1 + ΔS, S1 + 2ΔS, and at time T2 (T1 ≠ T2). Assume that the sample points of the second frame when imaged by the image sensor 1 are S2, S2 + ΔS, and S2 + 2ΔS. Further, at this time, it is assumed that the sample point S1 of the first frame and the sample point S2 of the second frame are misaligned due to camera shake or the like.

  Sample points S1, S1 + ΔS, and S1 + 2ΔS indicate sample points on the subject. In the low-resolution real image Fa that is the first frame shown in FIG. 2B, the sample points S1, S1 + ΔS, and S1 + 2ΔS are captured. The luminance values are the pixel values pa1, pa2, pa3 in the pixels P1, P2, P3. Sample points S2, S2 + ΔS, and S2 + 2ΔS indicate sample points on the subject. In the low-resolution real image Fb that is the second frame shown in FIG. 2C, the sample points S2, S2 + ΔS, and S2 + 2ΔS are captured. The luminance values are the pixel values pb1, pb2, and pb3 in the pixels P1, P2, and P3.

  Accordingly, the pixel values pa1, pa2, and pa3 of the pixels P1, P2, and P3 of the low-resolution real image Fa that becomes the first frame have the relationship as shown in FIG. 2B, and the low-resolution real image that becomes the second frame. The pixel values pb1, pb2, and pb3 in the pixels P1, P2, and P3 of the image Fb have a relationship as shown in FIG. As described above, the low-resolution real image Fa shown in FIG. 2B and the low-resolution real image Fb shown in FIG. 2C each have a pixel position of only (S1-S2) based on the position of the subject in the drawing. The image is shifted. When the low-resolution real image Fb is represented with reference to the pixels P1, P2, and P3 of the low-resolution real image Fa (that is, the low-resolution real image Fb is the amount of motion (S1-S2) relative to the low-resolution real image Fa). When the positional deviation is corrected), the low-resolution real image Fb is represented as shown in FIG.

  Then, as shown in FIG. 3A, the high-resolution image Fx1 is estimated by combining the low-resolution real images Fa and Fb as shown in FIGS. 2B and 2C with the positional deviation corrected. (STEP 31). At this time, in order to simplify the following description, for example, it is assumed that the resolution is doubled in the one-dimensional direction. That is, as pixels of the high-resolution image Fx1, in addition to the pixels P1, P2, and P3 of the low-resolution real images Fa and Fb, a pixel P4 that is located at an intermediate position between the pixels P1 and P2, and an intermediate position between the pixels P2 and P3 It is assumed that the pixel P5 located at is set.

  If the low-resolution actual image Fa is the reference frame, the pixel values at the pixels P1, P2, and P3 are set to the pixel values pa1, pa2, and pa3 in the low-resolution actual image Fa. For the pixel P4, the distance of the pixel position (center position of the pixel) from the pixel P4 is greater than the distance from the pixel positions of the pixels P1 and P2 in the low-resolution real image Fa to the pixel P1 in the low-resolution real image Fb. Since the distance from the pixel position is closer, the pixel value pb1 is assumed. Similarly, for the pixel P5, the distance between the pixel position (pixel center position) and the pixel P5 is smaller than the distance from the pixel positions of the pixels P2 and P3 in the low-resolution real image Fa, and the pixel P2 in the low-resolution real image Fb. It is assumed that the pixel value pb2 is obtained because the distance from the pixel position is shorter. As described above, a high resolution image obtained by setting the pixel values of the pixels P1 to P5 as pa1, pa2, pa3, pb1, and pb2 may be estimated as the high resolution image Fx1.

  Thereafter, the high-resolution image Fx1 obtained in STEP 31 is subjected to calculation using a conversion equation having parameters such as a down-sampling amount, a blur amount, and a positional shift amount (corresponding to a motion amount) as shown in FIG. ), The low-resolution estimated images Fa1 and Fb1 that are the estimated images for the low-resolution actual images Fa and Fb are generated (STEP 32). In FIG. 3B, low-resolution estimated images estimated by the n-th processing, that is, low-resolution estimated images Fan and Fbn corresponding to the low-resolution actual images Fa and Fb estimated from the high-resolution image Fxn are shown. Show.

  That is, based on the high-resolution image Fx1, the pixel values at the sample points S1, S1 + ΔS, S1 + 2ΔS are estimated, and the low-resolution estimated image Fa1 having the acquired pixel values pa11 to pa31 as the pixel values of the pixels P1 to P3 is generated. . Similarly, based on the high-resolution image Fx1, the pixel values at the sample points S2, S2 + ΔS, S2 + 2ΔS are estimated, and the low-resolution estimated image Fb1 that uses the acquired pixel values pb11 to pb31 as the pixel values of the pixels P1 to P3 is generated. To do.

  Then, as shown in FIG. 3C, the difference between the low resolution estimated images Fa1 and Fb1 thus obtained and the low resolution actual images Fa and Fb is obtained, and the differences are synthesized. Thus, the difference image ΔFx1 with respect to the high resolution image Fx1 is acquired (STEP 33). In FIG. 3C, the difference image ΔFxn with respect to the high-resolution image Fxn acquired by the n-th process, that is, the difference images ΔFan and ΔFbn between the low-resolution estimated images Fan and Fbn and the low-resolution actual images Fa and Fb. A difference image ΔFxn obtained by combining the images is shown.

  By the process of FIG. 3C, the difference (pa11−pa1), (pa21−pa2), and (pa31−pa3) between the pixels P1, P2, and P3 in the low resolution estimated image Fa1 and the low resolution actual image Fa. A difference image ΔFb1 is obtained from the differences (pb11−pb1), (pb21−pb2), and (pb31−pb3) at the pixels P1, P2, and P3 in the image ΔFa1, the low resolution estimated image Fa2, and the low resolution actual image Fb. That is, in the difference image ΔFa1, the difference values (pa11−pa1), (pa21−pa2), and (pa31−pa3) are pixel values, and in the difference image ΔFb1, the difference values (pb11−pb1), (pb21−pb2), (Pb31-pb3) is the pixel value.

  Then, by combining the pixel values of the difference images ΔFa1 and ΔFb1, the difference values in the pixels F1 to F5 are calculated, and the difference image ΔFx1 with respect to the high resolution image Fx1 is acquired. When obtaining the difference image ΔFx1 by combining the pixel values of the difference images ΔFa1 and ΔFb1, for example, in the ML method or the MAP method, a square error is used as an evaluation function. That is, the evaluation function of the ML method or the MAP method becomes a value obtained by squaring the pixel values of the difference images ΔFa1 and ΔFb1 and adding them between frames. Therefore, the gradient, which is the differential value of the evaluation function, is a value obtained by doubling the pixel values of the difference images ΔFa1 and ΔFb1, so that the difference image ΔFx1 with respect to the high resolution image Fx1 has the pixel values of the difference images ΔFa1 and ΔFb1. It is calculated by increasing the resolution using the doubled value.

  Thereafter, as shown in FIG. 3D, the pixel values (difference values) of the pixels P1 to P5 in the obtained difference image ΔFx1 are obtained from the pixel values of the pixels P1 to P5 in the high resolution image Fx1 estimated in STEP1. By subtracting, the high-resolution image Fx2 having a pixel value close to the subject having the luminance distribution shown in FIG. 2A is reconstructed (STEP 34). In FIG. 3D, the high-resolution image Fx (n + 1) obtained by the n-th processing, that is, the high-resolution image Fx (n + 1) obtained by subtracting the difference image ΔFxn from the high-resolution estimated image Fxn. Indicates.

  Then, by repeating the processing of STEP 32 to STEP 34 described above, the pixel value of the difference image ΔFxn obtained in STEP 33 is reduced, and the pixel value of the high-resolution image Fxn is close to the subject having the luminance distribution shown in FIG. Converges to pixel values. In STEP 32 and STEP 34 in the n-th process, the low-resolution estimated images Fan and Fbn and the high-resolution estimated image Fx (n + 1) are obtained by the high-resolution image Fxn obtained in STEP 34 in the previous (n−1) -th process. To be acquired. Then, when the pixel value (difference value) of the difference image ΔFxn becomes smaller than a predetermined value, or when the pixel value (difference value) of the difference image ΔFxn is converged, in the previous process (n-1th process). The super-resolution processing is terminated with the high-resolution image Fxn obtained in STEP 34 as the target high-resolution image.

  In each of the following embodiments of the present invention, when super-resolution processing is performed during moving image shooting, the number of repetitions of the above-described STEP 32 to STEP 34 is set to a small number, and super-resolution processing is performed during still image shooting. In the case of performing the above, the number of repetitions of the above STEP 32 to STEP 34 is set to a larger number than the time of moving image shooting. That is, for example, when the number of repetitions of the above-described STEP 32 to STEP 34 during moving image shooting is N1, the number of repetitions of the above-described STEP 32 to STEP 34 during still image shooting is N2, which is larger than N1. Further, at the time of moving image shooting, the number of repetitions N1 of the above-described STEP32 to STEP34 is set to 1, the above-described STEP32 to STEP34 are performed only once, and the high-resolution image Fx2 is estimated from the initial setting image Fx1. It does n’t matter.

  As the super-resolution processing based on the above-described flow, the ML method, the MAP method, the POCS method, the IBP method, and the like mentioned in [Background Art] are used in the image processing unit 4. For example, when the MAP method is used, the difference image ΔFxn with respect to the high-resolution image Fxn acquired in STEP 33 is configured by a gradient image having a pixel value as a value obtained by differentiating the evaluation function in the MAP method by the gradient method. To do.

  That is, the high-resolution image Fxn constrained by the posterior probability distribution is added to the differential image of the high-resolution image Fxn obtained by combining the differential images ΔFan and ΔFbn obtained by the least square method using the square error at each pixel position as the pixel value. After adding the high-speed image, the gradient image corresponding to the difference image ΔFxn in STEP 33 is acquired by performing differentiation using the gradient method. In STEP 34, the high-resolution image Fx (n + 1) is reconstructed by subtracting the difference image ΔFxn based on the gradient image from the high-resolution image Fxn.

  An embodiment of an imaging device including the image processing unit 4 that performs such super-resolution processing will be described below. In the following embodiments, the details of the super-resolution processing of the image processing unit 4 are characterized. The moving image shooting operation, the still image shooting operation, the moving image playback operation, and the still image playback operation are described above. Therefore, the detailed description thereof is omitted. In each of the following embodiments, the super-resolution operation by the image processing unit 4 will be described on behalf of the MAP method.

<First Embodiment>
A first embodiment of the present invention will be described with reference to the drawings. FIG. 4 is a block diagram illustrating a configuration of an image processing unit in the imaging apparatus according to the present embodiment.

  As shown in FIG. 4, the image processing unit 4 in the imaging apparatus according to the present embodiment includes a frame memory 41 that temporarily stores an image signal that is a low-resolution real image for one frame that has been converted into a digital signal by the AFE 2; Similarly, a frame memory 42 for temporarily storing a low-resolution real image for one frame, given a low-resolution real image for one frame stored in the frame memory 41, and a low-resolution for the current frame input from the AFE 2 A motion amount calculation unit 43 for calculating a motion amount (position shift amount) given an image signal based on an actual image and an image signal based on a low-resolution actual image one frame before the current frame stored in the frame memory 41; Based on the amount of motion detected by the amount calculation unit 43, super-resolution processing is performed on the low-resolution real images of two frames stored in the frame memories 41 and 42 to obtain a high A super-resolution processing unit 44 that generates an image image, a frame memory 45 that temporarily stores the high-resolution image obtained by the super-resolution processing unit 44, and supplies the high-resolution image to the super-resolution processing unit 44 again; And a signal processing unit 46 that generates a luminance signal and a color difference signal from an image signal based on a high-resolution image subjected to super-resolution processing in the image processing unit 44 or an image signal directly given from the AFE 2.

  Further, the super-resolution processing unit 44 generates an initial high-resolution image from two frames of low-resolution real images (corresponding to the low-resolution real images Fa and Fb in FIG. 3A) stored in the frame memories 41 and 42. The initial high resolution estimation unit 441 that estimates the high resolution image (corresponding to the high resolution image Fx1 in FIG. 3A), the high resolution image estimated by the initial high resolution estimation unit 441, and the frame memory 45 A selection unit 442 that selects any one of the temporarily stored high-resolution images and gives it to the subsequent stage; the high-resolution image selected by the selection unit 442; and the low-resolution for two frames stored in the frame memories 41 and 42 A high-resolution update amount calculation unit 443 that obtains an update amount of the high-resolution image (corresponding to the difference image ΔFxn in FIG. 3D) from the actual image, and a high-resolution update amount from the high-resolution image selected by the selection unit 442 It includes a subtracting unit 444 subtracts the update amount obtained by the detection section 443, a.

(Operation of the motion detection unit)
First, the operation of the motion amount detection unit 43 will be briefly described. First, the motion amount detection unit 43 is provided with the low-resolution real image Fa that is the current frame from the AFE 2 and the low-resolution real image Fb that is one frame before the current frame is provided from the frame memory 41, so Is detected. The detection of the amount of movement (positional shift amount) is performed by setting the current low-resolution real image Fa as a non-reference image out of two temporally continuous low-resolution real images Fa and Fb, and using the other low-resolution image. The actual image Fb is detected as a reference image. In the present embodiment, the other low-resolution actual image Fb that is not the current frame is described as the reference image. However, the low-resolution actual image Fa that is the current frame may be the reference image.

  For this motion amount detection, for example, the motion amount in units of pixels between images is detected by a well-known representative point matching method, and then, for the motion amount in one pixel, the positional displacement correction of pixel units is performed. It may be detected based on the relationship between pixel values. In the following, the representative point matching method and the positional deviation detection within one pixel will be briefly described.

(1) Representative Point Matching Method For each of the reference image and the non-reference image, for example, as shown in FIG. 5, an a × b pixel group (for example, a 36 × 36 pixel group) is set as one small region e. Further, the small region e group (for example, 6 × 8 small region e group) of the p × q region is further divided as one detection region E. In the reference image, in each small region e, as shown in FIG. 6A, one pixel is set as the representative point R from the a × b pixels constituting the small region e. In the non-reference image, In each small region e, as shown in FIG. 6B, a plurality of pixels among a × b pixels constituting the small region e are set as sampling points S (for example, all the pixels of a × b May be used as the sampling point S).

  Thus, when the small area e and the detection area E are set, the pixel value of each sampling point S of the non-reference image and the representative point R of the reference image for the small area e at the same position of the reference image and the non-reference image. A difference from the pixel value is obtained as a correlation value at each sampling point S. Then, for each detection region E, the correlation values of the sampling points S whose relative positions to the representative point R are the same among the small regions e are cumulatively added for all the small regions e constituting the detection region E. Thus, the cumulative correlation value at each sampling point S is acquired. Thus, for each detection region E, the correlation values of p × q sampling points S having the same relative position to the representative point R are cumulatively added, so that cumulative correlation values for the number of sampling points are obtained. (For example, when all the pixels of a × b are set as the sampling point S, a × b cumulative correlation values are obtained).

  In this way, when the cumulative correlation value for each sampling point S is obtained for each detection region E, the cumulative correlation value considered to have the highest correlation with the representative point R in each detection region E is the minimum. A sampling point S is detected. In each detection region E, the amount of motion between the sampling point S and the representative point R that minimizes the cumulative correlation value is obtained from each pixel position. Thereafter, by averaging the motion amounts obtained for each detection region E, this average value is detected as a motion amount in pixel units between the reference image and the non-reference image.

(2) Motion amount detection within one pixel As described above, when a motion amount in pixel units is detected, a motion amount within one pixel is further detected. At this time, for example, for each small region e shown in FIG. 5, first, the correlation between the representative point R and the representative point R is determined based on the pixel value of the pixel serving as the representative point R of the reference image and the amount of motion obtained by the above-described representative point matching method. The amount of motion in one pixel is detected based on the relationship between the pixel value of the high sampling point Sx and the pixel values of the surrounding pixels.

  That is, in each small region e, the pixel value La (see FIG. 7A) of the representative point R, which is the pixel position (ar, br) in the reference image, and the pixel position (as, bs) in the non-reference image. The pixel value Lb of the sample point Sx, the pixel value Lc (see FIG. 7B) of the pixel position (as + 1, bs) adjacent to the sample point Sx in the horizontal direction, and the pixel position adjacent to the sample point Sx in the vertical direction The amount of motion in one pixel is detected based on the relationship with the pixel value Ld (see FIG. 7B) of (as, bs + 1). At this time, according to the representative point matching method, the amount of movement of the pixel unit from the reference pixel to the non-reference image becomes a value represented by a vector amount of (as-ar, bs-br).

  Further, as shown in FIG. 8A, the pixel value Lb is changed linearly from the pixel value Lb by shifting one pixel in the horizontal direction from the pixel that is the sample point Sx, as shown in FIG. 8B. Furthermore, it is assumed that the pixel value Lb changes linearly from the pixel value Ld by shifting one pixel in the vertical direction from the pixel that becomes the sample point Sx. 7A and 7B, the horizontal position Δx (= (La−Lb) / (Lc−Lb) where the pixel value La is between the pixel values Lb and Lc. ) And a vertical position Δy (= (La−Lb) / (Ld−Lb)) that is the pixel value La between the pixel values Lb and Ld. That is, the vector amount represented by (Δx, Δy) is obtained as the amount of motion within one pixel between the reference pixel and the non-reference pixel.

  Thus, when the motion amount within one pixel in each of the small regions e is obtained, the obtained motion amount is averaged, and this average value is obtained as a pixel unit between the reference image Fb and the non-reference image Fa. It is detected as the amount of movement by. Then, the motion amount between the reference image Fb and the non-reference image Fa can be calculated by adding the obtained motion amount within one pixel to the motion amount in pixel units obtained by the representative point matching method. it can.

  The low resolution image that is optimal for the super-resolution processing can be selected based on the motion amount acquired by the motion amount detection by the above method, and the low resolution real image Fa stored in the frame memory 41 can be selected. Thus, it is possible to perform positional deviation correction based on the low-resolution real image Fb stored in the frame memory 42. The motion amount detection in the motion amount detection unit 43 has been described by taking the above method as an example. However, if the motion amount within the pixel can be detected, other methods can be used to detect the motion amount. You may make it detect the amount of motion between the low-resolution real images for a frame.

(Operation of super-resolution processor)
Next, the operation of the super-resolution processing unit 44 configured as described above will be described with reference to FIG. Note that u is the number of pixels of the low-resolution image including the low-resolution real image, and v is the number of pixels of the high-resolution image including the initial high-resolution image. Then, each pixel value of the low-resolution real image Fa using u pixels is Ya = [ya1, ya2,..., Au], and each pixel value of the low-resolution real image Fb using u pixels is Yb = [yb1, yb2,. ybu]. When super-resolution processing is performed in the image processing unit 4, the low-resolution real images Fa and Fb for two frames stored in the frame memories 41 and 42 are sent to the initial high-resolution estimation unit 441 and the high-resolution update amount calculation unit 443. Given.

(1) Processing by Initial High-Resolution Image In the initial high-resolution estimation unit 441 to which the low-resolution real images Fa and Fb are given, the motion obtained by the motion amount calculation unit 43 as in STEP 31 shown in FIG. Based on the amount, a positional shift between the low-resolution real images Fa and Fb of two frames is detected. Then, an initial high-resolution image Fx1 with v pixels is obtained by performing an interpolation process using a positional shift confirmed by the amount of motion on the pixel values Ya and Yb of the low-resolution real images Fa and Fb. It is assumed that the pixel values of the high resolution images Fx1 to Fxn are X = [x1, x2,..., Xv]. When the initial high-resolution image Fx1 is generated by the initial high-resolution estimation unit 441, a filter that performs noise reduction processing using a median filter or the like is used as a part of the high-resolution image generation filter, or a high-resolution image generation It is preferable to increase the S / N ratio (signal / noise ratio) by increasing the number of taps of the filter.

  When the initial high resolution image Fx1 is output from the initial high resolution estimation unit 441 as described above, the selection unit 442 selects the initial high resolution image Fx1 and provides it to the high resolution update amount calculation unit 443. In the high-resolution update amount calculation unit 443, the position of each of the low-resolution real images Fa and Fb with respect to the high-resolution real image Fx based on the amount of motion is given by the motion amount obtained by the motion amount calculation unit 43. Displacement and blur due to low resolution are calculated. Camera parameters including positional deviation and blur obtained from the amount of motion, and down-sampling amounts from v-pixel high-resolution image Fxn to low-resolution estimated images Fan and Fbn corresponding to u-pixel low-resolution actual images Fa and Fb Matrixes Wa and Wb are obtained.

Then, the initial high resolution image Fx1 given through the selection unit 442 is multiplied by the camera parameter matrices Wa and Wb, so that the low resolution real images Fa and Fb are obtained as in STEP32 shown in FIG. Low-resolution estimated images Fa1 (= Wa · X) and Fb1 (= Wb · X) are estimated. Further, the high-resolution update amount calculation unit 443 is provided with the low-resolution actual images Fa and Fb for two frames stored in the frame memories 41 and 42, so that the estimated low-resolution estimated images Fa1 and Fb1 and the low-resolution Differences | Wa · X−Ya | ² and | Wb · X−Yb | ² due to square errors with the actual images Fa and Fb are obtained.

The evaluation function I based on the difference between the low-resolution estimated images Fa1, Fb1 and the low-resolution actual images Fa, Fb is added with a constraint term obtained by the high-resolution image selected by the selection unit 442 to the added value of each difference. It is required in the form. That is, the obtained evaluation function I is | Wa · X−Ya | ² + | Wb · X−Yb | ² + γ | CX | ² . In the constraint term γ | CX | ² of the evaluation function I, the matrix C is a matrix based on the prior probability model, and based on the prior knowledge that “the high-resolution image has few high-frequency components”, for example, Laplacian It is composed of a high-pass filter such as a filter. The coefficient γ is a parameter representing the strength of the weight in the evaluation function I of the constraint term.

To perform a minimization with respect to the way the obtained evaluation function I by the gradient method, the gradient ∂I / ∂X for the evaluation function ^{I (= 2 × {Wa T} · (Wa · X-Ya) + Wb T · (Wb · X−Yb) + γC ^T · C · X}). The subscript “T” represents a transposed matrix. By obtaining the gradient ∂I / ∂X that is the update amount, the high-resolution update amount calculation unit 443 obtains the difference image ΔFx1 in STEP 33 shown in FIG.

  When the update amount based on the gradient ∂I / ∂X calculated in this way is given from the high resolution update amount calculation unit 443 to the subtraction unit 444, the subtraction unit 444 selects as shown in STEP 34 in FIG. The update amount by the gradient ∂I / ∂X is subtracted from the initial high-resolution image Fx1 provided from the initial high-resolution estimation unit 441 via the unit 442. As a result of the subtraction by the subtracting unit 444, a high-resolution image Fx2 in which the pixel values of the v pixels are (X−∂I / ∂X) is obtained. Then, the newly obtained high resolution image Fx2 is output from the subtracting unit 444, given to the frame memory 45, and temporarily stored.

(2) Processing by high-resolution image after generation As described above, the initial high-resolution image Fx1 is acquired, and the update amount of the high-resolution image is set as the gradient ∂I / ∂X, and a new high-resolution image Fx2 is acquired. Then, the selection unit 442 selects the high-resolution image stored in the frame memory 45 and gives it to the high-resolution update amount calculation unit 443. That is, in the case of the nth (n ≧ 2) arithmetic processing, the high-resolution image Fxn stored in the frame memory 45 is read by the selection unit 442 and provided to the high-resolution update amount calculation unit 443.

Accordingly, the high resolution update amount calculation unit 443 estimates the low resolution estimated images Fan (= Wa · X) and Fbn (= Wb · X) from the high resolution image Fxn stored in the frame memory 45. Then, differences | Wa · X−Ya | ² and | Wb · X−Yb | ² due to square errors in the low resolution actual images Fa and Fb and the low resolution estimated images Fan and Fbn are obtained. As a result, the gradient ∂I / に 対 す る X with respect to the evaluation function I is calculated as the update amount in the nth arithmetic processing.

That is, the estimated low-resolution images Fan, and Fbn, low-resolution real image Fa that is input by the Fb and high resolution image Fxn, the update amount ∂I / ∂X (= 2 × { Wa T · (Wa · X−Ya) + Wb ^T · (Wb · X−Yb) + γC ^T · C · X}) is obtained by the high-resolution update amount calculation unit 443. Then, in the subtraction unit 445, the difference image ΔFxn obtained by the high resolution update amount calculation unit 443 is subtracted from the high resolution image Fxn given from the frame memory 45 via the selection unit 442, that is, X−∂I When the calculation of / X is performed, a new high-resolution image Fx (n + 1) is obtained.

  Then, in the super-resolution processing unit 44, when the number of times of arithmetic processing by the high-resolution update amount calculation unit 443 is set to n times, the high-resolution image Fx (n + 1) newly obtained by the subtraction unit 445 is used. The signal is output from the subtracting unit 445 to the signal processing unit 46. Further, in the super-resolution processing unit 44, when the number of arithmetic processings by the high-resolution update amount calculation unit 443 is set to n + 1 or more, the high-resolution image Fx (n + 1) is given to the frame memory 45 and temporarily stored. Then, the (n + 1) th arithmetic processing is performed.

(Super-resolution processing operation during movie shooting)
In the imaging device in which the super-resolution processing unit 44 operates as described above, when a moving image shooting is instructed, the imaging processing time of one frame becomes short, so the super-resolution processing operation in the super-resolution processing unit 44 It is necessary to shorten the time required for the above. Therefore, in this embodiment, when super-resolution processing is requested during moving image shooting, the high-resolution image generation processing is generated only by the initial high-resolution image Fx1, and the processing time is set to a short time. Thereby, the storage operation of the high resolution image by the frame memory 45 can be omitted.

  That is, when the image signal output from the image sensor 1 is converted into a digital signal by the AFE 2 and applied to the image processing unit 4, the low-resolution real image Fa that becomes the current frame is stored in the frame memory 41. At this time, the low-resolution real image Fb of the previous frame stored in the frame memory 41 is stored in the frame memory 42. The low resolution actual image Fa from the AFE 2 is also supplied to the motion amount detection unit 43, and the motion amount with the low resolution actual image Fb stored in the frame memory 41 is detected.

  Then, the low-resolution real images Fa and Fb stored in the frame memories 41 and 42 and the motion amount detected by the motion detection unit 43 are given to the super-resolution processing unit 44 to perform super-resolution processing. Is done. In the super-resolution processing unit 44, the initial high resolution estimation unit 441 acquires the initial high resolution image Fx1 from the low resolution real images Fa and Fb based on the motion amount obtained by the motion amount calculation unit 43. When the initial high-resolution image Fx1 is given to the high-resolution update amount calculation unit 443 via the selection unit 442, a difference image ΔFx1 with the update amount ∂I / ∂X is obtained from the low-resolution actual images Fa and Fb. . Then, the subtraction unit 444 subtracts the update amount ∂I / ∂X based on the difference image ΔFx1 from the initial high resolution image Fx1, thereby obtaining the high resolution image Fx2.

  The high-resolution image Fx2 obtained by the super-resolution processing unit 44 in this way is output from the subtraction unit 444 to the signal processing unit 46, thereby generating a luminance signal and a color difference signal. The luminance signal and the color difference signal for the high resolution image generated by the signal processing unit 46 of the image processing unit 4 are given to the compression processing unit 6. The image signal based on the luminance signal and the color difference signal is compression-encoded by the MPEG compression encoding method together with the audio signal from the audio processing unit 5 to generate a compressed encoded signal for moving images.

(Super-resolution processing when shooting still images)
Also, when still image shooting is instructed, unlike the time of moving image shooting, the shooting processing time does not become short, so the time taken for the super-resolution processing operation in the super-resolution processing unit 44 is long. can do. Therefore, as in the case where super-resolution processing is requested during moving image shooting, the high-resolution image generation processing estimates the initial high-resolution image Fx1, and then repeats the setting processing for the high-resolution image using the frame memory 45.

  That is, after the high-resolution image generated by the super-resolution processing unit 44 is stored in the frame memory 45, an update amount is calculated based on the high-resolution image stored in the frame memory 45, and based on this update amount The operation of generating a new high resolution image is repeated a plurality of times. The number of iterations may be a preset number, or may be repeated until the update amount becomes smaller than a predetermined value.

  As in the case of moving image shooting, when the image signal output from the image sensor 1 is converted into a digital signal by the AFE 2 and given to the image processing unit 4, the low-resolution real image Fa that becomes the current frame is stored in the frame memory 41. At the same time, the low-resolution real image Fb of the previous frame stored in the frame memory 41 is stored in the frame memory 42. Further, the low-resolution real images Fa and Fb are supplied from the AFE 2 and the frame memory 41 to the motion amount detection unit 43, and the motion amount is calculated.

  When the low-resolution real images Fa and Fb stored in the frame memories 41 and 42 and the motion amount from the motion amount detection unit 43 are given to the super-resolution processing unit 44, the initial high-resolution estimation unit 441 An initial high resolution image Fx1 is estimated. Further, the update amount ∂I / ∂X between the initial high resolution image Fx1 and the low resolution actual images Fa and Fb is calculated by the high resolution update amount calculation unit 443, and subtracted from the initial high resolution image Fx1 by the subtraction unit 444. Thus, a high resolution image Fx2 is obtained.

  When the high resolution image Fx2 obtained by the super-resolution processing unit 44 in this way is given to the frame memory 45 and temporarily stored, the high resolution update amount calculation unit 443 is passed through the selection unit 442. Will be given. Thereafter, as described above, the high-resolution update amount calculation unit 443 uses the high-resolution image Fxn stored in the frame memory 45 and the low-resolution images Fa and Fb from the frame memories 41 and 42 to update the amount ∂I / A difference image ΔFxn by ∂X is calculated.

  Then, the subtraction unit 444 subtracts the update amount ∂I / ∂X based on the difference image ΔFxn from the high resolution image Fxn to obtain the high resolution image Fx (n + 1). At this time, when the number of calculation processing times n is a preset number of iterations, or when the update amount ∂I / ∂X calculated by the high-resolution update amount calculation unit 443 is equal to or less than a predetermined value. The acquired high resolution image Fx (n + 1) is output to the signal processing unit 46. Conversely, when these conditions are not satisfied, the acquired high-resolution image Fx (n + 1) is output to the frame memory 45 in order to repeat the arithmetic processing.

  When a high-resolution image obtained by repeatedly performing arithmetic processing in the super-resolution processing unit 44 in this manner is output to the signal processing unit 46, a luminance signal and a color difference signal based on the high-resolution image are generated. . The luminance signal and the color difference signal for the high resolution image generated by the signal processing unit 46 of the image processing unit 4 are given to the compression processing unit 6. The image signal based on the luminance signal and the color difference signal is subjected to compression encoding processing by the JPEG compression encoding method, and a compressed encoded signal for a still image is generated.

  In this embodiment, the high-resolution image generated by the super-resolution processing unit 44 is temporarily stored in the frame memory 45 at the time of moving image shooting, and the arithmetic processing for super-resolution is not repeated again. , And output to the signal processing unit 46. However, as in the case of still image shooting, the frame memory 45 may be temporarily stored and a high-resolution image obtained by repeating the arithmetic processing in the super-resolution processing unit 44 a plurality of times may be output. As described above, when the calculation process in the super-resolution processing unit 44 is repeated a plurality of times during moving image shooting, the number of repetitions is set to the number corresponding to the shooting period for each frame as described above, and the still image The number is set to be smaller than the number of repetitions at the time of shooting.

  In addition, the number of iterations of the calculation process in the super-resolution processing is reduced during moving image shooting compared to still image shooting. However, super-resolution is similarly applied during continuous shooting of a plurality of frames. It is also possible to reduce the number of iterations of arithmetic processing in image processing.

  Furthermore, in the present embodiment, the super-resolution processing unit 44 acquires a high-resolution image from low-resolution real images for two frames that are continuously captured. A high-resolution image may be generated. At this time, as many frame memories as the number of frames of the low-resolution real image used for generating the high-resolution image are required. In addition, a motion amount storage unit that stores the motion amount of the low-resolution real image detected every two consecutive frames is required by one less than the number of frames.

(Configuration for super-resolution processing with 3 frames)
Hereinafter, as a configuration example of the image processing unit 4 when a high-resolution image can be acquired from a low-resolution real image of 3 frames or more, a configuration in the case of using a 3-frame low-resolution real image is shown. FIG. 9 is a block diagram showing the configuration of the image processing unit of this example. In the configuration of the image processing unit shown in FIG. 9, the same parts as those in the configuration of FIG. 4 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The image processing unit 4 shown in FIG. 9 includes frame memories 41 and 42, a motion amount detection unit 43, a super-resolution processing unit 44, a frame memory 45, and a signal processing unit 46, as in the configuration of FIG. In addition, a frame memory 47 is provided for temporarily storing a low-resolution real image stored in the frame memory 42, and a motion amount detected by the motion amount detection unit 43 is provided for temporary storage. A movement amount storage unit 48.

  In such a configuration, when the low-resolution real image Fa that is the current frame is supplied from the AFE 2 to the image processing unit 4, the low-resolution real image Fb one frame before stored in the frame memory 41 is the frame memory 42. And the low-resolution real image Fc two frames before stored in the frame memory 42 is supplied to the frame memory 47. Further, the low-resolution actual image Fa from the AFE 2 is given to the motion amount detection unit 43, so that the motion amount between the low-resolution actual images Fa and Fb is detected.

  At this time, the motion amount between the low-resolution real images Fb and Fc already detected is given to the motion amount storage unit 48 and temporarily stored. Further, the motion amount detection unit 43 adds the motion amount between the low-resolution actual images Fb and Fc and the motion amount between the low-resolution actual images Fa and Fb, thereby moving the motion between the low-resolution actual images Fa and Fc. Calculate the amount. Then, the motion amount between the low-resolution actual images Fa and Fc detected by the motion amount detection unit 43 and the motion amount between the low-resolution actual images Fb and Fc stored in the motion amount storage unit 48 are super solutions. This is given to the image processing unit 44.

  Then, the low-resolution real images Fa, Fb, Fc stored in the frame memories 41, 42, 47 and the motion amounts from the motion amount detection unit 43 and the motion amount storage unit 48 are stored in the super-resolution processing unit 44. Given this, super-resolution processing is performed. In the super-resolution processing unit 44, the initial high-resolution estimation unit 441 uses three consecutive low-resolution real images Fa to Fc based on the motion amounts from the motion amount detection unit 43 and the motion amount storage unit 48. An initial high resolution image Fx1 is acquired.

  The high-resolution update amount calculation unit 443 also includes a difference constituted by the update amount ∂I / ∂X by the continuous three frames of low-resolution real images Fa to Fc and the high-resolution image selected by the selection unit 442. An image is required. As a result, the subtraction unit 444 subtracts the update amount ∂I / ∂X based on the difference image acquired by the high resolution update amount calculation unit 443 from the high resolution image selected by the selection unit 442. An image is generated. When the high-resolution image obtained by the super-resolution processing unit 44 in this way is output from the subtraction unit 444 to the signal processing unit 46, and the luminance signal and the color difference signal are generated, the compression processing unit 6 A compression encoding process is performed to generate a compression encoding signal for a moving image or a still image.

<Second Embodiment>
A second embodiment of the present invention will be described with reference to the drawings. FIG. 10 is a block diagram illustrating a configuration of an image processing unit in the imaging apparatus of the present embodiment. In the configuration of the image processing unit shown in FIG. 10, the same parts as those in the configuration of FIG. 9 are denoted by the same reference numerals and detailed description thereof is omitted.

  As shown in FIG. 10, the image processing unit 4 in the imaging apparatus of the present embodiment has a configuration in which the frame memory 47 in the configuration in FIG. 9 is omitted, and one frame of low resolution recorded in the frame memory 42. A selection unit 49 is provided that selects either the real image or the high-resolution image generated by the super-resolution processing unit 44 and supplies the selected image to the frame memory 45.

  In the imaging apparatus including the image processing unit 4 configured as described above, the selection unit 49 sets an image to be selected and output to the frame memory 45 depending on whether the shooting operation is a moving image shooting or a still image shooting. The That is, when moving image shooting is instructed, the selection unit 49 selects a low-resolution real image from the frame memory 42. On the other hand, when still image shooting is instructed, the selection unit 49 selects the high-resolution image generated by the super-resolution processing unit 44.

  Therefore, at the time of moving image shooting, the frame memory 45 is used as a frame memory (corresponding to the frame memory 47 in FIG. 9) for storing a low-resolution real image two frames before the current frame, and a super-resolution processing unit. At 44, as in the image processing unit 4 having the configuration shown in FIG. 9, a high-resolution image is generated from three consecutive low-resolution real images. At this time, the frame memory 45 is not used for returning the high-resolution image generated by the super-resolution processing unit 44 to the super-resolution processing unit 44 again. Therefore, the arithmetic processing in the super-resolution processing unit 44 is performed only once without being repeated.

  Because of this operation, when a low-resolution real image Fa is input from the AFE 2, three consecutive low-resolution real images Fa, Fb, and Fc are given to the frame memories 41, 42, and 45, respectively. Will be memorized. Then, the motion amount detection unit 43 gives the motion amount between the low resolution actual images Fb and Fc to the motion amount storage unit 48 for temporary storage, and detects the motion amount between the low resolution actual images Fa and Fb. . Further, the motion amount detection unit 43 adds the motion amount between the low-resolution actual images Fa and Fb and the motion amount between the low-resolution actual images Fb and Fc, thereby moving the motion between the low-resolution actual images Fa and Fc. Calculate the amount. Then, the low-resolution real images Fa to Fc from the frame memories 41, 42, and 45 and the motion amounts from the motion amount detection unit 43 and the motion amount storage unit 48 are given to the super-resolution processing unit 44.

  Therefore, in the super-resolution processing unit 44, the initial high-resolution estimation unit 441 (see FIG. 9) acquires the initial high-resolution image Fx1 from the continuous three frames of the low-resolution real images Fa to Fc, and updates the high resolution. In the amount calculation unit 443 (see FIG. 9), a difference image ΔFx1 composed of the update amount ∂I / ∂X is obtained from the initial high-resolution image Fx1 and the low-resolution real images Fa to Fc. The initial high resolution image Fx1 and the difference image ΔFx1 obtained in this way are given to the subtraction unit 444 (see FIG. 9), whereby the high resolution image Fx2 to be output to the signal processing unit 46 is acquired.

  On the other hand, at the time of still image shooting, the frame memory 45 is used as a frame memory (corresponding to the frame memory 45 in FIG. 4) for storing the high-resolution image generated by the super-resolution processing unit 44. In the image processing unit 44, similarly to the image processing unit 4 configured as shown in FIG. 4, a high-resolution image is generated from two consecutive low-resolution real images. Therefore, the high-resolution image generated by the super-resolution processing unit 44 is fed back to the input side of the super-resolution processing unit 44 by the frame memory 45, and the arithmetic processing in the super-resolution processing unit 44 is repeatedly performed. It becomes.

  Because of this operation, when a low-resolution real image Fa is input from the AFE 2, two consecutive low-resolution real images Fa and Fb are given to the frame memories 41 and 42 and temporarily stored. The Rukoto. Then, the motion amount detection unit 43 detects the motion amount between the low-resolution real image and the low-resolution real images Fa and Fb. Then, the low-resolution real images Fa and Fb from the frame memories 41 and 42 and the motion amount from the motion amount detection unit 43 are given to the super-resolution processing unit 44.

  Therefore, in the super-resolution processing unit 44, first, the initial high-resolution estimation unit 441 (see FIG. 4) acquires the initial high-resolution image Fx1 from the low-resolution real images Fa and Fb of two consecutive frames. In the resolution update amount calculation unit 443 (see FIG. 4), a difference image ΔFx1 composed of the update amount ∂I / ∂X is obtained from the initial high-resolution image Fx1 and the low-resolution real images Fa and Fb. The initial high resolution image Fx1 and the difference image ΔFx1 obtained in this way are given to the subtraction unit 444 (see FIG. 4), whereby the high resolution image Fx2 is acquired and temporarily stored in the frame memory 45.

  Thereafter, the update amount 処理 I / ∂X calculated by the high-resolution update amount calculation unit 443 (see FIG. 4) is a predetermined value until the number of calculation processes in the super-resolution processing unit 44 reaches the set number of iterations. The operation of generating the high-resolution image Fx (n + 1) from the high-resolution image Fxn and the low-resolution real images Fa and Fb is performed until the following. Then, when the number of calculation processing times n is a preset number of iterations, or the update amount ∂I / ∂X calculated by the high-resolution update amount calculation unit 443 (see FIG. 4) is equal to or less than a predetermined value. When it becomes, the acquired high resolution image Fx (n + 1) is output to the signal processing unit 46.

  As described above, in this embodiment, the frame memory 45 is used as a frame memory for storing a low-resolution real image at the time of moving image shooting, and also as a frame memory for storing a high-resolution image at the time of still image shooting. can do. Therefore, at the time of moving image shooting, a high-resolution image with high reproducibility can be generated by increasing the number of frames of the low-resolution real image used for the super-resolution processing. On the other hand, at the time of still image shooting, by repeating the calculation in the super-resolution processing, a high-resolution image with high reproducibility can be generated from a low-resolution real image with a small number of frames.

  In this embodiment, a high-resolution image is generated by three consecutive low-resolution real images during moving image shooting. However, the high-resolution image may be generated by four or more low-resolution real images. . At this time, the number of frame memories is the same as the number of frames of the low-resolution real image used for the super-resolution processing at the time of moving image shooting, and the frame of the low-resolution real image used for the super-resolution processing at the time of moving image shooting. Super-resolution processing at the time of still image shooting is realized by a low-resolution real image that is one frame less than the number.

<Third Embodiment>
A third embodiment of the present invention will be described with reference to the drawings. The configuration of the image processing apparatus in the imaging apparatus of the present embodiment is the same as that shown in the block diagram of FIG. 4 as in the first embodiment. In this embodiment, the configuration is the same as that of the first embodiment, but it may be the same as the configuration of the second embodiment.

  Unlike the first embodiment, the image processing unit 4 in the imaging apparatus of the present embodiment has an edge region that contains a large amount of edge components and a small number of edge components when the difference image ΔFxn is obtained in STEP 33 in the operation flow of FIG. A flat region is determined, and a coefficient corresponding to the edge component is multiplied by the difference image ΔFxn for each region. That is, the calculation in STEP 34 is performed so that the obtained difference image ΔFxn is reflected in the edge region, and the calculation in STEP 34 is performed in the flat region so that the influence of the obtained difference image ΔFxn is reduced. Made.

  Therefore, in the following, the detection of the edge region and the flat region is performed in this way, and the super-resolution processing operation part considering the state of each detected region, that is, the calculation operation of the difference image ΔFxn will be described in detail. It is assumed that the description of the same operation part in the first embodiment is omitted.

Similar to the image processing unit 4 in the first embodiment, in the super-resolution processing unit 44, the initial high-resolution estimation unit 441 receives the low-resolution real images Fa and Fb, thereby acquiring the initial high-resolution image Fx1. The The high-resolution update amount calculation unit 443 evaluates the high-resolution actual image Fxn (n ≧ 1 and includes the initial high-resolution image Fx1) selected by the selection unit 442 and the low-resolution actual images Fa and Fb. gradient for the function I ∂I / ∂X (= 2 × {Wa T · (Wa · X-Ya) + Wb T · (Wb · X-Yb) + γC T · C · X}) is obtained.

  Furthermore, when the gradient ∂I / ∂X is obtained in this way, in the present embodiment, by further confirming the image components in the low resolution actual images Fa and Fb and the low resolution estimated images Fan and Fbn, For each of the low-resolution real images Fa and Fb, a region that is estimated to contain a lot of noise and a region that has a low correlation between edge portions are confirmed. For example, by confirming the correlation between the image components of the low-resolution actual image Fa and the low-resolution estimated image Fan, a flat portion where noise is easily confirmed in each region in the low-resolution actual image Fa, and erroneously detecting noise The ratio with the edge part including the edge to be detected is detected. Similarly, the flat part and the edge part in the low-resolution actual image Fb are detected by confirming the correlation between the image components of the low-resolution actual image Fb and the low-resolution estimated image Fbn.

  Then, for each of the low-resolution actual image and the low-resolution estimated image, the correlation with the component ratio between the edge portion and the flat portion estimated in each region (for example, the correlation between the image components of the low-resolution actual image and the low-resolution estimated image) Accordingly, a weighting coefficient k to be given to the difference due to the square error between the low-resolution actual image and the low-resolution estimated image at the gradient ∂I / ∂X is set. At this time, the weight coefficient k is determined for a region that is estimated to have a low correlation with the component ratio between the edge portion and the flat portion (a region in which there is no correlation between the image components of the low-resolution actual image and the low-resolution estimated image). A value close to 0 is set. On the other hand, for a region that is estimated to have a high correlation with the component ratio between the edge portion and the flat portion (region where the correlation between the image components of the low-resolution actual image and the low-resolution estimated image is large), the weighting factor k is A value close to 1 is set.

  That is, for a region that is estimated to have a low correlation with the component ratio between the edge portion and the flat portion, the difference due to the square error between the low-resolution actual image and the low-resolution estimated image in the gradient ∂I / ∂X. The weighting factor k is set so as to reduce the influence of. This weighting factor k is set for each pixel of each of the low resolution real images Fa and Fb. The weighting factor k is a value satisfying 0 ≦ k ≦ 1, and when the weighting factor k is a value close to 1, a difference (gradient ∂I / ∂X) including an edge component when the high-resolution image is updated. ). Further, when the weight coefficient is a value close to 0, the value of the initial high resolution image with reduced noise is used as it is when the high resolution image is updated.

  Therefore, from the relationship between the image components of the pixel value Ya of the low-resolution actual image Fa and the pixel value Wa · X of the low-resolution estimated image Fan, the coefficient for the low-resolution actual image Fa is Ka = [ka1, ka2,. ]. Similarly, from the relationship between the image components of the pixel value Yb of the low resolution actual image Fb and the pixel value Wb · X of the low resolution estimated image Fbn, the coefficient for the low resolution actual image Fb is Kb = [kb1, kb2,. kbu].

In this way, when the coefficients Ka and Kb are set for the low-resolution real images Fa and Fb, the high-resolution update amount calculation unit 443 uses the set coefficients as (Wa · X−Ya) in the gradient ∂I / ∂X. , (Wb · X-Yb) by multiplying the gradient considering noise ∂J / ∂X (= 2 × { Wa T · Ka · (Wa · X-Ya) + Wb T · Kb · (Wb · X −Yb) + γC ^T · C · X}) is calculated as the update amount of the high resolution image. By obtaining the gradient ∂J / ∂X as the update amount, the high-resolution update amount calculation unit 443 obtains the difference image ΔFxn in STEP 33 shown in FIG.

  When the update amount based on the gradient ∂J / ∂X calculated in this way is given from the high resolution update amount calculation unit 443 to the subtraction unit 444, the subtraction unit 444 selects as shown in STEP 34 in FIG. The update amount by the gradient ∂J / よ り X is subtracted from the high-resolution image Fxn given from the unit 442. As a result of the subtraction by the subtracting unit 444, a high-resolution image Fx (n + 1) in which each pixel value of the v pixel is (X−∂J / ∂X) is obtained. Then, the newly obtained high-resolution image Fx (n + 1) is output from the subtraction unit 444 and provided to the frame memory 45 or the signal processing unit 46.

  As described above, the super-resolution processing unit 44 performs arithmetic processing according to the noise content estimated in each region in the low-resolution real image, so that in the high-resolution image obtained by the super-resolution processing, Noise can be reduced. Then, the number of iterations of the arithmetic processing in the super-resolution processing unit 44 at the time of moving image shooting is set to be smaller than that at the time of still image shooting, and processing can be performed within the shooting processing time of one frame. At this time, for example, as described in the first and second embodiments, the high-resolution image Fx2 set from the initial high-resolution image Fx1 may be output to the signal processing unit 46 without repeating. Absent. Further, at the time of still image shooting, when the number of arithmetic processing times n in the super-resolution processing is a preset number of iterations, or the update amount ∂J calculated by the high-resolution update amount calculation unit 443 When ∂X is equal to or less than a predetermined value, the acquired high resolution image Fx (n + 1) is output to the signal processing unit 46.

  In the above-described configuration, a high-resolution image is generated using two consecutive low-resolution real images during moving image shooting. However, the high-resolution image may be generated using three or more low-resolution real images. Furthermore, as a configuration similar to the configuration of the second embodiment, a frame memory 45 that temporarily stores a high-resolution image generated by the super-resolution processing unit 44 is supplied from the AFE 2 during supervision and is super-resolution. The image processing unit 44 may be used as a frame memory for storing a low-resolution real image used for generating a high-resolution image.

  Further, a detailed example of the arithmetic processing for estimating the correlation between the noise of each region and the component ratio of the edge in the low-resolution real image, which is performed by the high-resolution update amount calculation unit 443 of the super-resolution processing unit 44, This will be described below.

(First example of calculation processing for noise content estimation)
A first example of details of calculation processing for estimating the noise content of each region in the low-resolution actual image in the high-resolution update amount calculation unit 443 will be described. In this example, a high-pass filter (HPF) (not shown) that passes a high-frequency component of the high-resolution image Fxn selected by the selection unit 442 in the high-resolution update amount calculation unit 443 and low-frequency signals from the frame memories 41 and 42, respectively. An HPF (not shown) that allows high-frequency components of the resolution real images Fa and Fb to pass therethrough is provided.

  The high-resolution update amount calculation unit 443 further acquires high-frequency components of the low-resolution estimated images Fan and Fbn from the high-frequency components of the high-resolution image Fxn. Then, based on the correlation between the high-frequency components of the low-resolution estimated images Fan and Fbn and the high-frequency components of the low-resolution actual images Fa and Fb, the square error between the low-resolution actual image and the low-resolution estimated image Coefficients Ka and Kb to be given to the difference are set.

  That is, the high-frequency component HX (= [hx1, hx2,..., Hxv]) of the v-pixel high-resolution image Fxn is extracted by the HPF (not shown) installed in the high-resolution update amount calculation unit 443, and the u-pixel low The high frequency components HYa (= [hya1, hy2,..., Hyau]) of the resolution real image Fa are extracted, and the high frequency components HYb (= [hyb1, hyb2,..., Hybu]) of the u resolution low resolution real image Fb are extracted. Is done. The high frequency component HX of the high resolution image Fxn is multiplied by camera parameter matrices Wa and Wb. As a result, the high-frequency components Wa · HX (= [whxa1, whxa2,... Whxau]) of the u-pixel low-resolution estimated image Fax and the high-frequency components Wb · HX (= [whxb1, whxb2, ..., whxbu]).

  Thereafter, a high-frequency component ratio matrix Ra (= [ra1, ra2,..., Ra]) at the same pixel position is obtained for the high-frequency components HYa, Wa.Hx of the low-resolution actual image Fa and the low-resolution estimated image Fax. Similarly, a high frequency component ratio matrix Rb (= [rb1, rb2,..., Rbu]) at the same pixel position is obtained for the high frequency components HYb, Wb, and Hx of the low resolution actual image Fb and the low resolution estimated image Fbx. . At this time, a value rai that is an element in the ratio matrix Ra is calculated by the following equation (1), and a value rbi that is an element in the ratio matrix Rb is calculated by the following equation (2). Note that i in the expressions (1) and (2) is an integer satisfying 1 ≦ i ≦ u.

  The ratio Ra thus calculated is treated as a correlation value between the low resolution actual image Fa and the low resolution estimated image Fan. That is, for the i-th pixel in which the value rai in the ratio Ra is close to the predetermined value th1, the low-resolution actual image Fa and the low-resolution estimated image Fan have a strong correlation, and an area that includes many edge portions. Confirm that it is a pixel. Therefore, the weight coefficient kai for this i-th pixel is determined to be a value close to 1. On the other hand, for the i-th pixel in which the value rai in the ratio Ra is a value away from the predetermined value th1, the correlation between the low-resolution actual image Fa and the low-resolution estimated image Fan is weak, and the flatness where noise is easily confirmed It is confirmed that the pixel is in a region where there is a region or an edge portion that includes a large portion of the structure but the correlation between the low-resolution actual image and the low-resolution estimated image cannot be obtained. Therefore, the weight coefficient kai for this i-th pixel is determined to be a value close to zero. As described above, the larger the difference between the value rai and the predetermined value th1 in the ratio Ra, the closer the value of the weighting factor kai is to 0, and the larger the difference between the value rai and the predetermined value th1 in the ratio Ra is. The coefficient matrix Ka (= [ka1, ka2,..., Kau]) for the u-resolution low-resolution real image Fa is set so that the smaller the value, the value of the weighting coefficient kai approaches 1.

  Similarly, the ratio Rb calculated as described above is treated as a correlation value between the low-resolution actual image Fb and the low-resolution estimated image Fbn, the value rbi in the ratio Rb is compared with the predetermined value th1, and this i th The value of the weighting coefficient kbi for the pixels is determined. Therefore, the larger the difference between the value rbi and the predetermined value th1 in the ratio Rb, the closer the value of the weighting factor kbi is to 0, and the smaller the difference between the value rbi and the predetermined value th1 in the ratio Rb. The coefficient matrix Kb (= [kb1, kb2,..., Kbu]) for the u-resolution low-resolution real image Fb is set so that the value of the weighting coefficient kbi approaches 1 as the value becomes.

  Note that the relationship between the values rai and rbi in the ratios Ra and Rb and the weighting coefficients kai and kbi in the coefficient matrices Ka and Kb is expressed as shown in FIG. In the following, for simplicity of explanation, as shown in FIG. 11, the values rai and rbi are set to a common r, and the weighting factors kai and kbi are set to a common k. That is, when the predetermined value th1 is 1 ≦ th1, the relationship between the high-frequency component ratio r and the weighting coefficient k is as follows. In addition, when the same amount of edge component is included in the low-resolution actual image and the low-resolution estimated image, the low-resolution estimated image becomes a blurred image. Since Ra and Rb values rai and rbi are 1 or more, the predetermined value th1 is set to 1 ≦ th1.

(1) In the case of 0 ≦ | r−th1 | ≦ t0 (0 ≦ t0 ≦ 1-th1) The value of the weighting factor k is k = 1.
(2) When t0 <| r−th1 | ≦ t1 (t0 <t1) The value of the weighting factor k is k = t0 + 1− | r−th1 |.
(3) In the case of t1 <| r−th1 | The value of the weighting factor k is k = 0.

In this way, by u pixels of the coefficient matrix Ka, Kb are calculated, the gradient considering noise ∂J / ∂X (= 2 × { Wa T · Ka · (Wa · X-Ya) + Wb T Kb · (Wb · X−Yb) + γC ^T · C · X}) can be calculated as the update amount of the high-resolution image. Therefore, it is possible to reduce the influence of the differential value due to the square error in the region including many flat portions having a large noise content in each of the low-resolution real images Fa and Fb on the obtained update amount (gradient ∂J / ∂X). it can.

  As described above, in this example, the HPF is used to extract the high-frequency components of the low-resolution real image and the low-resolution estimated image, and the edge portion and the flat portion in the low-resolution real image are extracted according to the ratio of the high-frequency components. And the ratio can be detected. The amount of update for the pixel in the region that includes the component determined to be noise or is estimated to have low correlation in the edge portion by setting the portion where the frequency of the high-frequency component in each image is the same as the edge portion. And high-resolution image noise obtained by super-resolution processing can be suppressed.

(Second example of calculation processing for noise content estimation)
A second example of the details of the calculation processing for estimating the noise content of each region in the low-resolution actual image in the high-resolution update amount calculation unit 443 will be described. In this example, unlike the first example, the high-resolution update amount calculation unit 443 is used to check the correlation between the low-resolution actual image and the low-resolution estimated image in order to calculate the coefficient matrices Ka and Kb. Standard deviation is used as an image component. Therefore, the calculation operation of the coefficient matrices Ka and Kb based on the correlation between the low-resolution actual image and the low-resolution estimated image based on the standard deviation will be described in detail below.

  In this example, the high resolution update amount calculation unit 443 calculates standard deviations of the high resolution real image Fxn and the low resolution real images Fa and Fb. That is, the standard deviation of each pixel in the image is obtained by using pixel values of a plurality of pixels including a pixel of interest for obtaining the standard deviation and a plurality of peripheral pixels adjacent to the periphery of the pixel of interest. Therefore, for example, when 9 pixels arranged in a 3 × 3 matrix are used, as shown in FIG. 12, when obtaining the standard deviation σ for the pixel of interest G00, a 3 × 3 matrix in which the pixel of interest G00 is arranged in the center. Arithmetic processing according to the following expression (3) is performed on the pixel values z (-1) (-1) to z11 of the arranged pixels G (-1) (-1) to G11. In the equation (3), the pixel value of the pixel Gij is zij, and the average of the pixel values of the pixels G (-1) (-1) to G11 is z.

  Therefore, the standard deviation of the v pixel with respect to each pixel is obtained by performing the above-described arithmetic processing on the pixel value X (= [x1, x2,..., Xv]) of each pixel of the high resolution image Fxn of v pixels. SX (= [sx1, sx2,..., Sxv]) is calculated. Then, the standard deviation Wa · SX (= [wsxa1, wsxa2,..., Wsxau]) of the low resolution estimated image Fan and the low resolution are obtained by multiplying the standard deviation SX by the camera parameter matrix Wa, Wb. A standard deviation Wb · SX (= [wsxb1, wsxb2,..., Wsxbu]) of the estimated image Fbn is obtained.

  In addition, the above-described arithmetic processing is performed on the pixel value Ya (= [ya1, ya2,..., Yau]) of each pixel in the u-pixel low-resolution real image Fa, so that the standard for each pixel of the u pixel. The deviation Sya (= [sya1, sya2,... Syau]) is calculated. Similarly, the above-described arithmetic processing is performed on the pixel value Yb (= [yb1, yb2,..., Ybu]) of each pixel in the low-resolution actual image Fb of u pixels, so that A standard deviation SYb (= [syb1, syb2,..., Sybu]) is calculated.

  Then, as in the first example, the standard deviation ratio matrix Ea (= [ea1, ea2,..., Eau) at the same pixel position based on the standard deviations Sya, Wa and Sx of the low resolution actual image Fa and the low resolution estimated image Fax, respectively. ]), And the standard deviation ratio matrix Eb (= [eb1, eb2,..., Ebu] at the same pixel position based on the standard deviations SYb, Wb and Sx of the low-resolution actual image Fb and the low-resolution estimated image Fbx, respectively. ) Is required. At this time, a value eai as an element in the ratio matrix Ea is calculated by the following equation (4), and a value ebi as an element in the ratio matrix Eb is calculated by the following equation (5). Note that i in the expressions (4) and (5) is an integer satisfying 1 ≦ i ≦ u.

  Then, like the ratio Ra in the first example, the ratio Ea is treated as a correlation value between the low resolution actual image Fa and the low resolution estimated image Fan. That is, the value eai at the ratio Ea is compared with the predetermined value th1 to determine the value of the weight coefficient kai for the i-th pixel. Therefore, the larger the difference between the value ea at the ratio Ea and the predetermined value th1, the closer the value of the weighting factor kai is to 0, and the smaller the difference between the value ei at the ratio Eb and the predetermined value th1. The coefficient matrix Ka (= [ka1, ka2,..., Kau]) for the u-resolution low-resolution actual image Fa is set so that the value of the weighting coefficient kai approaches 1.

  Similarly, the ratio Eb is also treated as a correlation value between the low resolution actual image Fb and the low resolution estimated image Fbn, the value ebi in the ratio Eb is compared with the predetermined value th1, and the weighting coefficient for the i th pixel is calculated. Determine the value of kbi. Therefore, the larger the difference between the value ebi and the predetermined value th1 at the ratio Eb, the closer the value of the weighting factor kbi approaches 0, and the smaller the difference between the value ebi and the predetermined value th1 at the ratio Eb. The coefficient matrix Kb (= [kb1, kb2,..., Kbu]) for the u-resolution low-resolution real image Fb is set so that the value of the weighting coefficient kbi approaches 1 as the value becomes.

  Note that the relationship between the values ea, ebi at the ratios Ea, Eb and the weighting coefficients kai, kbi in the coefficient matrices Ka, Kb is expressed as shown in FIG. In the following, for ease of explanation, as shown in FIG. 13, the values ea and ebi are assumed to be common e, and the weighting factors kai and kbi are assumed to be common k. That is, when the above-mentioned predetermined value th1 is 1 ≦ th1 (for the same reason as in the first embodiment, the predetermined value th1 is set to 1 ≦ th1), it is the same as the relationship between the high-frequency component ratio r and the weighting factor k. The relationship between the standard deviation ratio e and the weighting coefficient k is as follows.

(1) When 0 ≦ | e−th1 | ≦ t0 (0 ≦ t0 ≦ 1-th1) The value of the weighting factor k is k = 1.
(2) When t0 <| e−th1 | ≦ t1 (t0 <t1) The value of the weighting factor k is k = t0 + 1− | e−th1 |.
(3) When t1 <| e−th1 | The value of the weighting factor k is k = 0.

  When the coefficient matrices Ka and Kb for the u pixels are calculated, the update amount ∂J / ∂X to be given to the subtracting unit 444 is obtained by performing arithmetic processing as in the first example. As described above, in this example, the ratio between the edge portion and the flat portion in the low-resolution actual image is detected based on the ratio of the standard deviation calculated for each of the low-resolution actual image and the low-resolution estimated image. Then, by setting the portion where the standard deviation in each image is equivalent to the edge portion, the amount of update for the pixels in the region that includes the component determined to be noise or is estimated to have low correlation in the edge portion is suppressed. In addition, it is possible to suppress noise in a high-resolution image obtained by super-resolution processing.

(Third example of calculation processing for noise content estimation)
A third example of the details of the calculation processing for estimating the noise content of each region in the low-resolution actual image in the high-resolution update amount calculation unit 443 will be described. In this example, unlike the first example and the second example, the correlation between the low-resolution actual image and the low-resolution estimated image is confirmed in the high-resolution update amount calculation unit 443 in order to calculate the coefficient matrices Ka and Kb. An edge direction component is used as an image component used at this time. Therefore, the calculation operation of the coefficient matrices Ka and Kb based on the correlation between the low-resolution actual image and the low-resolution estimated image based on the edge direction component will be described in detail below.

  In this example, the high-resolution update amount calculation unit 443 is provided with a plurality of types of edge direction filters corresponding to each edge direction component for each of the high-resolution real image Fxn and the low-resolution real images Fa and Fb. In the following description, the horizontal edge direction filter for calculating the edge direction component in the horizontal direction and the vertical edge direction for calculating the edge direction component in the vertical direction are respectively the high resolution real image Fxn and the low resolution real images Fa and Fb. Shall be provided.

  First, the horizontal edge direction filter and the vertical edge direction filter will be briefly described below. In the horizontal edge direction filter and the vertical edge direction filter, pixel values of a plurality of pixels including a pixel of interest for obtaining edge direction components in the horizontal direction and the vertical direction and a plurality of peripheral pixels adjacent to the periphery of the pixel of interest. Is used to determine the edge direction components of the horizontal and vertical directions of each pixel in the image. In the following, for example, pixel values z (-1) (-1) to z11 of 9 pixels G (-1) (-1) to G11 arranged in a 3 * 3 matrix shown in FIG. The description will be made on the assumption that the edge direction components in the horizontal direction and the vertical direction are calculated by being used. At this time, an edge direction component for the target pixel G00 is obtained.

  First, in the horizontal edge direction filter, pixels G (-1) (-1), G0 (-1), G0 (-1), G0 (-1), G0 (-1), The pixel value z (-1) (-1), z0 (-1), z1 (-1) of G1 (-1) is multiplied by "-1", and in the vertical direction with respect to the target pixel G00. The pixel values z (-1) 1, z01, and z11 of the pixels G (-1) 1, G01, and G11 arranged on the lower side are multiplied by "1", and the pixel value G00 is perpendicular to the target pixel G00. The value obtained by multiplying the pixel values z (-1) 0, z00, z10 of the pixels G (-1) 0, G00, G10 arranged at the same position by "0" is the horizontal direction. Edge component h. That is, the horizontal edge component h is calculated by the equation (6).

  Therefore, the above-described arithmetic processing is performed on the pixel value X (= [x1, x2,..., Xv]) of each pixel in the v-pixel high-resolution image Fxn by such a horizontal edge direction filter. , Horizontal edge direction component OhX (= [ohx1, ohx2,..., Ohxv]) for each pixel of v pixels is calculated. Then, by multiplying the edge direction component OhX by the camera parameter matrix Wa, Wb, the horizontal edge direction component Wa · OhX (= [wohxa1, wohxa2,..., Wohxau]) of the low resolution estimated image Fan. , And the horizontal edge direction component Wb · OhX (= [woxb1, whxb2,..., Whxbu]) of the low resolution estimated image Fbn is obtained.

  Then, the same arithmetic processing by the horizontal edge direction filter is performed on the pixel value Ya (= [ya1, ya2,..., Yau]) of each pixel in the low-resolution real image Fa of u pixels. A horizontal edge direction component OhYa (= [ohya1, ohya2,..., Ohyau]) for each pixel is calculated. Similarly, the same arithmetic processing by the horizontal edge direction filter is performed on the pixel value Yb (= [yb1, yb2,..., Ybu]) of each pixel in the low-resolution real image Fb of u pixels, A horizontal edge direction component OhYb (= [ohyb1, ohyb2,..., ohybu]) for each pixel of the u pixel is calculated.

  In the vertical edge direction filter, pixels G (-1) (-1) and G (-1) 0 arranged on the left side in the horizontal direction with respect to the target pixel G00 by the filter shown in FIG. , G (-1) 1 pixel values z (-1) (-1), z (-1) 0, and z (-1) 1 are multiplied by "-1" to obtain the target pixel G00. The pixel values z1 (-1), z10, and z11 of the pixels G1 (-1), G10, and G11 arranged on the right side in the horizontal direction are multiplied by “1”, and the horizontal direction is applied to the target pixel G00. The values obtained by multiplying the pixel values z0 (-1), z00, z01 of the pixels G0 (-1), G00, G01 arranged at the same position by “0” in the vertical direction The edge component is p. That is, the edge component p in the vertical direction is calculated by the equation (7).

  Therefore, the vertical edge direction with respect to each pixel of the v pixel is obtained by performing the above-described arithmetic processing on the pixel value X of each pixel in the high resolution image Fxn of the v pixel by such a vertical edge direction filter. The component OpX (= [opx1, opx2,..., Opxv]) is calculated. Then, by multiplying the edge direction component OpX by camera parameter matrices Wa and Wb, the vertical edge direction component Wa · OpX (= [wopxa1, wopxa2, ..., wopxau]) of the low resolution estimated image Fan is obtained. , And the vertical edge direction component Wb · OpX (= [wopxb1, wopxb2,..., Wopxbu]) of the low-resolution estimated image Fbn is obtained.

  Then, the same arithmetic processing by the vertical edge direction filter is performed on the pixel value Ya (= [ya1, ya2,..., Yau]) of each pixel in the low-resolution real image Fa of u pixels. A vertical edge direction component OpYa (= [opya1, opya2,..., Opyau]) of each pixel is calculated. Similarly, the same arithmetic processing by the vertical edge direction filter is performed on the pixel value Yb (= [yb1, yb2,..., Ybu]) of each pixel in the low-resolution real image Fb of u pixels, A vertical edge direction component OpYb (= [opyb1, opyb2,..., opybu]) of each u pixel is calculated.

  When the edge direction components in the horizontal direction and the vertical direction are calculated in this way, the edge components Wa, OhX, Wa in the horizontal direction and the vertical direction at the same pixel position in the low resolution actual image Fa and the low resolution estimated image Fax, respectively. The edge component ratio matrix Da (= [da1, da2,..., Dau]) is obtained from OpX, OhYa, OpYa. Similarly, the edge component ratio matrix Db (=) by the horizontal and vertical edge components Wb.OhX, Wb.OpX, OhYb, OpYb at the same pixel position in each of the low resolution actual image Fb and the low resolution estimated image Fbx. [Db1, db2,..., Dbu]). At this time, a value dai which is an element in the ratio matrix Da is calculated by the following equation (8), and a value dbi which is an element in the ratio matrix Db is calculated by the following equation (9). Note that i in the equations (8) and (9) is an integer satisfying 1 ≦ i ≦ u.

  Then, like the ratio Ra of the first example, the ratio Da is treated as a correlation value between the low-resolution actual image Fa and the low-resolution estimated image Fan. That is, the value dai at the ratio Da is compared with the predetermined value th1 to determine the value of the weight coefficient kai for the i-th pixel. Accordingly, the larger the difference between the value dai and the predetermined value th1 in the ratio Da, the closer the value of the weighting factor kai is to 0, and the smaller the difference between the value dai and the predetermined value th1 in the ratio Db. The coefficient matrix Ka (= [ka1, ka2,..., Kau]) for the u-resolution low-resolution actual image Fa is set so that the value of the weighting coefficient kai approaches 1.

  Similarly, the ratio Db is also treated as a correlation value between the low-resolution actual image Fb and the low-resolution estimated image Fbn, the value dbi in the ratio Db is compared with the predetermined value th1, and the weight coefficient for the i-th pixel Determine the value of kbi. Therefore, the larger the difference between the value dbi in the ratio Db and the predetermined value th1, the closer the value of the weighting factor kbi is to 0, and the smaller the difference between the value dbi in the ratio Db and the predetermined value th1. The coefficient matrix Kb (= [kb1, kb2,..., Kbu]) for the u-resolution low-resolution real image Fb is set so that the value of the weighting coefficient kbi approaches 1 as the value becomes.

  Note that the relationship between the values dai and dbi in the ratios Da and Db and the weighting coefficients kai and kbi in the coefficient matrices Ka and Kb is expressed as shown in FIG. In the following, for simplicity of explanation, as shown in FIG. 15, the values dai and dbi are assumed to be common d, and the weighting factors kai and kbi are assumed to be common k. That is, when the above-mentioned predetermined value th1 is 1 ≦ th1 (for the same reason as in the first embodiment, the predetermined value th1 is set to 1 ≦ th1), it is the same as the relationship between the high-frequency component ratio r and the weighting factor k. The relationship between the edge direction component ratio d and the weighting factor k is as follows.

(1) When 0 ≦ | d−th1 | ≦ t0 (0 ≦ t0 ≦ 1-th1) The value of the weighting factor k is k = 1.
(2) When t0 <| d−th1 | ≦ t1 (t0 <t1) The value of the weighting factor k is k = t0 + 1− | d−th1 |.
(3) In the case of t1 <| d−th1 | The value of the weighting factor k is k = 0.

  When the coefficient matrices Ka and Kb for the u pixels are calculated, the update amount ∂J / ∂X to be given to the subtracting unit 444 is obtained by performing arithmetic processing as in the first example. Thus, in this example, the ratio between the edge portion and the flat portion in the low-resolution actual image is detected based on the ratio of the edge direction component calculated for each of the low-resolution actual image and the low-resolution estimated image. Then, by updating the portion of each image where the value of the edge direction component is the same as the edge portion, the update is performed on the pixel in the region that includes the component determined to be noise or the edge portion is estimated to have low correlation. The amount of noise can be suppressed, and noise in a high-resolution image obtained by super-resolution processing can be suppressed.

  In this example, the edge direction component to be detected is described by taking the horizontal direction and the vertical direction as examples. However, the edge direction component is not limited to the horizontal direction and the vertical direction, and the edge direction component in another direction may be detected. . Furthermore, by detecting a plurality of edge direction components in different directions including the horizontal direction and the vertical direction, the ratio between the edge portion and the flat portion can be detected more accurately.

  Further, in each of the above-described embodiments, when the high-resolution image generation is performed in the super-resolution processing unit 44 with respect to the positional deviation correction between the frames in which the super-resolution processing is performed in the super-resolution processing unit 44, The description has been made assuming that the movement amount is based on the movement amount from the movement amount detection unit 43 and the movement amount storage unit 48. However, the positional deviation correction unit for each frame for performing positional deviation correction between the frames may be installed outside the super-resolution processing unit 44.

  That is, as in the configuration of FIG. 4, when the frame memories 41 and 42 are provided and two consecutive low-resolution images Fa and Fb are given to the super-resolution processing unit 44 to obtain the high-resolution image Fx, As in the configuration illustrated in FIG. 16, the position shift correction unit 50 is added to the configuration in FIG. 4. When the motion amount between the two consecutive low-resolution images Fa and Fb is detected by the motion amount detection operation of the motion amount detection unit 43, the position shift correction unit 50 is provided with the detected motion amount.

  Then, the positional deviation correction unit 50 is provided with the low-resolution real image Fa of the current frame stored in the frame memory 41, and the motion calculated by the motion amount detection unit 43 with respect to the low-resolution real image Fa. A positional deviation correction is performed based on the amount. As a result, the pixel position of the low-resolution actual image Fa is corrected using the low-resolution actual image Fb one frame before stored in the frame memory 42 as a reference. That is, the pixel position of the low-resolution actual image Fa is changed to a position based on the pixel position of the reference low-resolution actual image Fb. The low-resolution real image Fa that has been subjected to the positional shift correction by the positional shift correction unit 50 in this way is given to the super-resolution processing unit 44 together with the low-resolution real image Fb from the frame memory 42, and the high-resolution image Fx is converted into the high-resolution image Fx. Generated.

  Further, as in the configuration of FIG. 9, the frame memory 41, 42, 47 is provided, and three consecutive low-resolution images Fa to Fc are given to the super-resolution processing unit 44 to acquire the high-resolution image Fx. In this case, as in the configuration shown in FIG. 17, the configuration in which the position shift correction units 50 and 51 are added to the configuration in FIG. 9 is obtained. At this time, the motion amount between the low-resolution real images Fa and Fc detected by the motion amount detection unit 43 is given to the positional deviation correction unit 50 and the low-resolution real images Fb and Fb stored in the motion amount storage unit 48 The amount of motion between Fc is given to the positional deviation correction unit 51.

  Thereby, the positional deviation correction unit 50 is given the low-resolution real image Fa stored in the frame memory 41, and based on the motion amount between the low-resolution real images Fa and Fc acquired by the motion amount detection unit 43, A positional deviation correction based on the low resolution real image Fc is performed. Further, the positional deviation correction unit 51 is provided with the low resolution actual image Fb stored in the frame memory 42, and based on the motion amount between the low resolution actual images Fb and Fc acquired by the motion amount storage unit 48, A positional deviation correction is performed with the resolution actual image Fc as a reference. The low-resolution real images Fa and Fb that have been subjected to the positional shift correction by the positional shift correction units 50 and 51 in this way are supplied to the super-resolution processing unit 44 together with the low-resolution real image Fc from the frame memory 47, and the high-resolution processing unit 44 A resolution image Fx is generated.

  Further, when the selection unit 49 is provided as shown in FIG. 10 to select and store either the low resolution real image or the high resolution image in the frame memory 45, as shown in FIG. The configuration shown in FIG. 10 is obtained by adding position shift correction units 50 and 51 and a selection unit 52. As in the configuration of FIG. 17, the positional deviation correction unit 50 performs positional deviation correction on the low-resolution real image Fa stored in the frame memory 41 based on the motion amount acquired by the motion amount detection unit 43. The Further, the positional deviation correction unit 51 performs positional deviation correction on the low-resolution actual image Fb stored in the frame memory 42 based on the motion amount acquired by the motion amount storage unit 48.

  At the time of moving image shooting, the selection unit 49 selects the low resolution real image Fb from the frame memory 42 and outputs it to the frame memory 45, and the selection unit 52 receives the low resolution from the positional deviation correction unit 51. The actual image Fb is selected. Further, in the positional deviation correction unit 50, the amount of motion between the low resolution actual images Fa and Fc is detected by the motion amount detection unit 43, so that the low resolution actual image Fc is used as a reference for the low resolution actual image Fa. Misalignment correction is performed. Furthermore, since the amount of motion between the low resolution actual images Fb and Fc is stored in the motion amount storage unit 48 in the positional deviation correction unit 51, the low resolution actual image Fb is used as a reference for the low resolution actual image Fb. The positional deviation correction is performed.

  On the other hand, at the time of still image shooting, the selection unit 49 selects the high-resolution image Fx from the super-resolution processing unit 44 and outputs it to the frame memory 45, and the selection unit 52 selects the low-resolution image Fx from the frame memory 42. The resolution actual image Fb is selected. In the positional deviation correction unit 50, the amount of motion between the low resolution actual images Fa and Fb is detected by the motion amount detection unit 43, so that the low resolution actual image Fb is used as a reference for the low resolution actual image Fa. Misalignment correction is performed.

  In the above description, the image processing method according to the present invention has been described using the image pickup apparatus having the configuration shown in FIG. 1 as an example. However, the image processing method is not limited to the image pickup apparatus, and digital images such as liquid crystal displays and plasma televisions are used. The image processing method according to the present invention can also be used in a display device that performs processing. FIG. 19 shows a display device provided with an image processing device (corresponding to an “image processing unit”) that performs the image processing method of the present invention.

  The display device shown in FIG. 19 is similar to the image pickup device shown in FIG. 1 in that the image processing unit 4, the expansion processing unit 8, the display unit 9, the audio output circuit unit 10, the speaker unit 11, the timing generator 12, the CPU 13, the memory 14, An operation unit 15 and bus lines 16 and 17 are provided. A tuner unit 21 that selects a broadcast signal received externally, a demodulator unit 22 that demodulates a broadcast signal selected by the tuner unit 21, and an interface to which a compressed signal that is a digital signal input from the outside is input. 23.

  In the case of receiving a broadcast signal, the display device of FIG. 19 selects a broadcast signal of a desired channel by the tuner unit 21 and then demodulates the broadcast signal by the demodulation unit 22, so that MPEG compression is performed at the time of moving image reproduction. A digital signal which is a compressed signal by the encoding method is obtained. In addition, when a still image is reproduced, a digital signal that is a compressed signal by the JPEG compression code method is obtained. When this digital signal is given to the decompression processing unit 8, the digital signal which is a compressed signal is subjected to decompression processing by the MPEG compression code method or the JPEG compression code method.

  Then, when the operation unit 15 instructs to increase the resolution of the image, the image signal obtained by the expansion processing by the expansion processing unit 8 is given to the image processing unit 4 to select the above-described low-resolution real image. By performing the processing and the super-resolution processing, a high resolution image is generated. Thereafter, an image signal based on the generated high-resolution image is given to the display unit 9, and image reproduction is performed. Further, when reproducing a moving image, the audio signal obtained by the expansion processing of the expansion processing unit 8 is given to the speaker unit 11 through the audio output circuit unit 10 so that the sound is reproduced and output.

  The present invention can be applied to an imaging device or a display device that includes an image processing device that increases the resolution of an image by super-resolution processing.

These are the block diagrams which show the internal structure of the imaging device provided with the image process part used as the image processing apparatus of this invention. These are diagrams showing the relationship between the luminance distribution of a subject and image data in frames with different shooting times. These are the figures for demonstrating the outline | summary of the flow in a super-resolution process. These are block diagrams which show the structure of the super-resolution process part of the imaging device which becomes the 1st Embodiment of this invention. These are figures which show the relationship of the detection area in a representative point matching method. These are figures which show the relationship between the representative point of a reference | standard image, and the sampling point of a non-reference | standard image. FIG. 5 is a diagram illustrating a relationship between a representative point of a reference image and a sampling point of a non-reference image when position shift detection within one pixel is performed. FIG. 5 is a diagram illustrating a relationship between pixel values of a representative point of a reference image and a sampling point of a non-reference image when position shift detection within one pixel is performed. These are block diagrams which show another example of a structure of the super-resolution process part of the imaging device which becomes the 1st Embodiment of this invention. These are block diagrams which show the structure of the super-resolution process part of the imaging device which becomes the 2nd Embodiment of this invention. These are the graphs which show the relationship between the weighting factor and the ratio of a high frequency component in the 1st example of the arithmetic processing with respect to noise content estimation. These are figures for demonstrating the arithmetic processing operation | movement at the time of the standard deviation calculation made in the 2nd example of the arithmetic processing with respect to noise content estimation. These are graphs which show the relationship between the weighting factor and the ratio of the standard deviation in the second example of the arithmetic processing for noise content estimation. These are the figures for demonstrating the arithmetic processing operation | movement by the edge direction filter comprised in the 3rd example of the arithmetic processing with respect to noise content estimation. These are graphs which show the relationship between the weighting factor and the ratio of the edge direction in the third example of the arithmetic processing for noise content estimation. These are block diagrams which show another example of a structure of the super-resolution process part of the imaging device which becomes the 1st Embodiment of this invention. These are block diagrams which show another example of a structure of the super-resolution process part of the imaging device which becomes the 1st Embodiment of this invention. These are block diagrams which show another example of a structure of the super-resolution process part of the imaging device which becomes the 2nd Embodiment of this invention. These are block diagrams which show the internal structure of the display apparatus provided with the image processing part used as the image processing apparatus of this invention.

Explanation of symbols

1 Image sensor 2 AFE
Reference Signs List 3 microphone 4 image processing unit 5 audio processing unit 6 compression processing unit 7 driver unit 8 expansion processing unit 9 display unit 10 audio output circuit unit 11 speaker unit 12 timing generator 13 CPU
DESCRIPTION OF SYMBOLS 14 Memory 15 Operation part 16, 17 Bus line 20 External memory 41, 42, 45, 47 Frame memory 43 Motion amount calculation part 44 Super-resolution processing part 46 Signal processing part 48 Motion amount storage part 49,52 Selection part 50,51 Position shift correction unit

Claims (9)

In an image processing apparatus including a high resolution image generation unit that generates a high resolution image from a plurality of frames of low resolution images,
A plurality of first frame memories for temporarily storing the low resolution images;
A second frame memory that temporarily stores the high-resolution image generated by the high-resolution image generation unit;
With
The high-resolution image generation unit;
Generating the high resolution image based on the low resolution image of a plurality of frames stored in the first frame memory;
After the generated high resolution image is output to the second frame memory, the high resolution image stored in the second frame memory is used to perform an iterative calculation process for updating the high resolution image,
When generating and outputting the high-resolution image of a single frame, the number of iterations of the iterative calculation process using the second frame memory in the high-resolution image generation unit is a first number of times,
When continuously generating and outputting the high-resolution image of a plurality of frames, the number of iterations of the iterative calculation process using the second frame memory in the high-resolution image generation unit is less than the first number, and An image processing apparatus characterized in that the second number of times is zero or more. A plurality of the first frame memories are connected in series, and the continuously input low resolution image is stored for each frame.
The image processing apparatus according to claim 1, wherein the low-resolution images stored in the plurality of first frame memories are output in parallel to the high-resolution image generation unit. A third frame memory that is disposed downstream of the first frame memory and is commonly used as the first and second frame memories;
A selection unit that selects the low resolution image from the last stage of the first frame memory and the high resolution image generated by the high resolution image generation unit, and outputs the selected high resolution image to the third frame memory;
With
When generating and outputting the high resolution image of a single frame, the selection unit selects the high resolution image from the high resolution image generation unit and outputs the high resolution image to the third frame memory,
When continuously generating and outputting the high-resolution images of a plurality of frames, the selection unit selects the low-resolution images from the last stage of the first frame memory and outputs them to the third frame memory. The image processing apparatus according to claim 2. The high-resolution image generation unit;
Generating a high-resolution image from the plurality of low-resolution real images, and a high-resolution processing unit including the second frame memory as a part thereof;
A low-resolution estimation unit that acquires a plurality of low-resolution estimated images by estimating to be equivalent to each of the plurality of low-resolution actual images from the high-resolution image acquired by the high-resolution processing unit;
An update amount calculation unit that calculates an update amount of the high resolution image generated by the high resolution processing unit based on a difference value between the low resolution estimation image and the low resolution actual image obtained by the low resolution estimation unit; ,
With
In the high-resolution part, the high-resolution image is updated by updating the high-resolution image generated in the previous process with the update amount obtained in the update amount calculation unit. The image processing apparatus according to any one of claims 1 to 3. The update amount calculation unit
A low-resolution update amount calculation unit that calculates a low-resolution update amount at each pixel position of the low-resolution actual image by obtaining a difference value from the corresponding low-resolution estimated image for each of the low-resolution actual images. ,
For each of the low-resolution real images, image components included in the pixel values of the low-resolution real image and the low-resolution estimated image at the same pixel position are compared, and based on the comparison result of the image components, the low-resolution real image A weighting factor calculation unit for generating a weighting factor for multiplying the resolution update amount;
After the low resolution update amount calculated by the low resolution update amount calculation unit is multiplied by the weighting factor calculated by the weighting factor calculation unit, based on the low resolution update amount multiplied by the weighting factor A high-resolution update amount calculation unit that calculates the high-resolution update amount according to the number of pixels of the high-resolution image;
With
In the high resolution processing unit, the high resolution image generated by the high resolution update amount calculated by the high resolution update amount calculation unit is updated with respect to the high resolution image generated in the previous process. The image processing apparatus according to claim 4, wherein the image processing apparatus is updated. The comparison result of the image components in the weight coefficient calculation unit is a ratio of image components obtained by dividing the image component of the pixel value of the low-resolution actual image by the image component of the pixel value of the low-resolution estimated image. And
When the ratio of the image components becomes a value close to a predetermined value of 1 or more, the weight coefficient is set to a value close to 1,
The image processing apparatus according to claim 5, wherein the weight coefficient is set to a value close to 0 when the ratio of the image components becomes a value farther than the predetermined value. In an electronic device provided with an image signal based on an image that becomes a plurality of frames by external input or imaging, and having a high resolution function that converts an image based on the image signal into a high resolution image,
The image processing unit according to any one of claims 1 to 6 is provided as an image processing unit that realizes the high resolution function, and the high resolution processing is performed using the image based on the image signal as the low resolution image. An electronic apparatus, wherein the desired high-resolution image is generated. A first step of generating a high resolution image based on a plurality of frames of low resolution images, and a second step of generating an update amount of the high resolution image generated in the first step, wherein the first step and An image processing method for updating the high-resolution image generated in the first step by an update amount in the second step by repeatedly performing the second step,
When generating and outputting the high-resolution image of a single frame, the number of iterations of the iterative calculation process by the first step and the second step is set as the first number of times,
When the high-resolution images of a plurality of frames are continuously generated and output, the number of iterations of the iterative calculation process in the first step and the second step is less than the first number and more than zero. An image processing method characterized in that the number of times is two. When generating and outputting the high-resolution image of a single frame, the number of frames of the low-resolution image used for generating the high-resolution image in the first step is a first frame number,
When continuously generating and outputting the high-resolution image of a plurality of frames, the number of frames of the low-resolution image used for generating the high-resolution image in the first step is greater than the first frame number. The image processing method according to claim 8, wherein the number of frames is two.

Images