JP2008294950A - Image processing method and device, and electronic device with the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method and device where a low resolution image is selected by determining a reference position when selecting a low resolution image used for obtaining a high resolution image and by referencing the relation between a pixel position of the low resolution image and reference position. <P>SOLUTION: By allowing the middle position between each pixel of the real low resolution image Fa forming a remarkable frame to be a reference position, the real low resolution image of the frame close to a sample point (pixel position) is selected as a real low resolution image so that super-resolution processing is carried out. Further, a high resolution image Fx is obtained by selecting the real low resolution images Fa, Fc and Fd comprised of three frames from the real low resolution images Fa-Fd comprised of four frames and by carrying out super-resolution processing therefor. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing method and an image processing apparatus for generating a high resolution image, and more particularly to an image processing method and an image processing apparatus for performing a super-resolution process for generating a high resolution image from a plurality of low resolution images. The present invention also relates to an electronic device that records or reproduces a high resolution image generated by the image processing apparatus.

  In recent years, with the development of various digital technologies, imaging devices such as digital cameras and digital videos that take digital images with solid-state imaging devices such as CCD (Charge Coupled Device) and CMOS (Complimentary Metal Oxide Semiconductor) sensors, Display devices such as liquid crystal displays and plasma televisions for displaying are becoming widespread. An image processing technique for converting the resolution of a digital image processed by such an imaging apparatus or display apparatus into a high resolution using a plurality of digital images taken at different times has been proposed.

  As this high-resolution conversion method, super-resolution processing for estimating one high-resolution image from a plurality of low-resolution images with positional deviation has been proposed (see Non-Patent Document 1). In this super-resolution processing, the generated high-resolution image is inversely transformed to estimate the original low-resolution image for constructing the high-resolution image, and then compared with the actual low-resolution image. In addition, a method called a reconstruction type that generates a high-resolution image close to an actual value has been proposed.

  In the super-resolution processing by the reconstruction type method, first, a high resolution image acquired from a plurality of low resolution images is assumed (STEP 1). Based on the assumed high resolution image, estimation is performed by performing inverse transform to each of a plurality of low resolution images for constructing the high resolution image (STEP 2). Then, after comparing the low-resolution image after the inverse conversion and the original low-resolution image (STEP 3), the comparison result shows that the pixel position between the converted low-resolution image and the original low-resolution image is the same at each pixel position. A high-resolution image is generated so that the difference in values is small (STEP 4). Then, the processing of STEP 2 to STEP 4 is repeated so that the difference in the value at each pixel position (hereinafter referred to as “pixel value”) between the converted low-resolution image and the original low-resolution image converges. Thus, the finally obtained high resolution image can be made an image close to the real image.

  As this reconstruction type method, ML (Maximum-Likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection Onto Convex Set) method, IBP (Iterative Back Projection) method, etc. are proposed. In the ML method, a pixel value of a low resolution image (hereinafter referred to as “low resolution estimated image”) estimated from a high resolution image and a pixel value of an actual low resolution image (hereinafter referred to as “low resolution actual image”) Is a method for generating a high-resolution image that minimizes the evaluation function (see Non-Patent Document 2). That is, the super-resolution processing by the ML method is a processing method based on the principle of maximum likelihood estimation.

  In the MAP method, a high-resolution image that minimizes the evaluation function is obtained by adding the probability information of the high-resolution image to the square error of the pixel values of the low-resolution estimated image and the low-resolution real image. It is the method of producing | generating (refer nonpatent literature 3, 4). That is, in the MAP method, an optimal high resolution image is acquired by estimating a high resolution image when the appearance probability in the posterior probability distribution based on the foresight information for the low resolution real image is maximized.

  Furthermore, in the POCS method, simultaneous equations relating to the pixel values of the high-resolution image and the low-resolution image are created, and the simultaneous equations are sequentially solved to obtain the optimum value of the pixel value of the high-resolution image. This is a method for generating a resolution image (see Non-Patent Document 5). The IBP method is a method for generating a high resolution from a plurality of low resolution images that have different shooting positions but have overlapping pixel positions of a subject, that is, a super-resolution processing method using an iterative backprojection method (non-resolution method). (See Patent Document 6).

In the super-resolution processing as described above, a plurality of low-resolution images that are continuous in time series are used to acquire a high-resolution image. Then, an image processing method has been proposed in which a low-resolution image that is optimal for acquiring this high-resolution image is selected from a plurality of low-resolution images and super-resolution processing is performed (see Patent Document 1). In the image processing method in Patent Document 1, motion information related to the motion of an image generation device such as an imaging device is added to image data of a low resolution image, and suitable for generating a high resolution image based on this motion information. It is determined whether or not. Further, the motion amount of the pixel in each image frame constituting the low resolution image is calculated from the pixel value, and based on this motion amount, it is determined whether or not it is suitable for generating the high resolution image.
JP 2006-41603 A Sung, CP Min, KP Moon, GK "Super-resolution image reconstruction: a technical overview", (Signal processing Magazine, IEEE on Volume 26, Issue 3, May 2003, P.21-36) Tom, BC Katsaggelos, AK "Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images", (Image Processing, Proceedings, International Conference on Volume 2, Oct 1995, P.23-26 ) "Extraction of high-resolution frames from video sequences" by Schultz, RR Stevenson, RL (Image Processing, IEEE Transactions on Volume 5, Issue 6, June 1996, P.996-1011) Hardie RC Barnard KJ Armstrong EE "Joint MAP registration and high-resolution image estimation using a sequence of undersampled images" (Image Processing, IEEE Transactions on Volume 6, Issue 12, Dec. 1997 P.1621-1633) Stark, H. Oskoui, P. "High resolution image recovery from image-plane arrays, using convex projections" (J. Opt. Soc. Am. A, on vol. 6. P.1715-1726, 1989) Michel, I. Peleg, S. "Improving Resolution by Image Registration" (CVGIP: Graph Models Image Process, on vol. 53. P.231-239, Mar. 1991)

  However, in the image processing method disclosed in Patent Document 1, it is necessary to check motion information added to image data of a low-resolution image when selecting an image frame that is a low-resolution image for generating a high-resolution image. Therefore, it cannot be applied when motion information is not added to the image data. Furthermore, depending on the amount of motion corresponding to the amount of positional deviation between images, pixel values near the interpolation pixel position in a high-resolution image can often be acquired. In the case of setting whether to permit or not, an image frame of a low-resolution image having such a motion amount may not be selected. Therefore, the efficiency for generating a high-resolution image may deteriorate.

  In view of such problems, the present invention sets a reference position for selection when selecting a low-resolution image to be used for acquiring a high-resolution image, and sets the pixel position and the reference position of the low-resolution image. It is an object to provide an image processing method and an image processing apparatus that select a low-resolution image based on the relative relationship between the image and the image processing apparatus. Further, according to the present invention, when selecting a low resolution image to be used for acquiring a high resolution image, a reference position for selection is set, and the relative position between the pixel position of the low resolution image and the reference position is low. It is an object of the present invention to provide an electronic apparatus including an image processing apparatus that selects a resolution image.

  In order to achieve the above object, an image processing apparatus according to the present invention generates a high resolution image from a first low resolution image that is a frame of interest and a second low resolution image that is an M (an integer greater than or equal to 1) frame. In the image processing apparatus including the conversion unit, a motion amount detection unit that detects a motion amount between each of the third low resolution images of N (an integer satisfying N> M) frames and the first low resolution image; A reference position set based on the pixel position of the first low-resolution image from the third low-resolution image of N frames in consideration of the amount of movement in the third low-resolution image obtained by the motion amount detection unit An image evaluation unit that selects, as the second low-resolution image, the third low-resolution image having an M frame closer in pixel position to the second low-resolution image by the image evaluation unit. The third Resolution image is given to the resolution increasing section, wherein the high-resolution image is generated.

  The reference position may be set to a position that is an intermediate position between adjacent pixel positions in the vertical direction and the horizontal direction in the first low-resolution image, and the reference position may be set in the high-resolution image. The pixel position may be set as an interpolation pixel for the first low-resolution image. Further, when the third low resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the position is an intermediate position between adjacent pixel positions in the vertical direction and the horizontal direction in the first low resolution image. The third low-resolution image whose pixel position is close to the image may be selected as the second low-resolution image.

  In such an image processing apparatus, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames in the image evaluation unit, the horizontal position between the pixel position and the reference position is set. The third low resolution image having a short distance in the direction may be selected as the second low resolution image.

  Further, in the image evaluation unit, when the third low resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the third low resolution with a small amount of motion with respect to a temporally adjacent previous frame. An image may be selected as the second low-resolution image.

  Further, in the image evaluation unit, when the third low resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the resolution enlargement ratio in each of the horizontal direction and the vertical direction is determined in the super-resolution step. When they are different, the low-resolution image having a short distance in the direction in which the enlargement ratio between the pixel position and the reference position is large may be selected as the second low-resolution image.

  In such an image processing apparatus, for each of the third low resolution images selected from the frame memory as the second low resolution image by the image evaluation unit, based on the motion amount obtained by the motion amount detection unit. And a positional deviation correction unit that corrects the pixel position on the basis of the first low-resolution image.

  The electronic apparatus according to the present invention is an electronic apparatus provided with an image signal based on an image that becomes a plurality of frames by external input or imaging, and having a high resolution function for converting an image based on the image signal into a high resolution image. As an image processing unit that realizes the high resolution function, any of the above-described image processing devices is provided, and the desired high resolution image is generated by performing high resolution processing using the image based on the image signal as the low resolution image. It is characterized by being.

  The image processing method according to the present invention further includes an image processing step including a resolution increasing step for generating a high resolution image from the first low resolution image serving as the frame of interest and the second low resolution image including M (an integer greater than or equal to 1). In the method, a motion amount detection step of detecting a motion amount between each of the third low resolution images of N (an integer satisfying N> M) frames and the first low resolution image, and the motion amount detection step In consideration of the amount of motion in the third low-resolution image, the pixel position is determined from the third low-resolution image of N frames with respect to a reference position set based on the pixel position of the first low-resolution image. A low-resolution image selection step of selecting the third low-resolution image of M frames closer as the second low-resolution image, and in the high-resolution step, the low-resolution image And generating the high-resolution image based and the third low-resolution image and the second low-resolution image in the selection step and the first low-resolution images.

  In such an image processing method, in the low resolution image selection step, the reference position is set to a position that is an intermediate position between adjacent pixel positions in the vertical direction and the horizontal direction in the first low resolution image. Alternatively, the reference position may be set to a pixel position that becomes an interpolation pixel for the first low-resolution image in the high-resolution image. In the low-resolution image selection step, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, they are adjacent in the vertical direction and the horizontal direction in the first low-resolution image. The third low-resolution image whose pixel position is close to a position that is an intermediate position of the pixel positions may be selected as the second low-resolution image.

  In the low-resolution image selection step, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the horizontal distance between the pixel position and the reference position is short. The third low-resolution image may be selected as the second low-resolution image, and the third low-resolution image having a small amount of motion with respect to the temporally adjacent previous frame may be selected as the second low-resolution image. It doesn't matter if you choose.

  Further, in the low-resolution image selection step, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the resolution is expanded in each of the horizontal direction and the vertical direction in the super-resolution step. When the rates are different, the third low-resolution image having a short distance in the direction in which the enlargement rate between the pixel position and the reference position is large may be selected as the second low-resolution image.

  According to the present invention, a low-resolution image with a large amount of motion is not unselected as in the prior art, but a reference position is set and a low-resolution image whose pixel position is close to the reference position is selected. Thereby, when selecting a low resolution image other than the low resolution image that is the frame of interest in order to generate a high resolution image, it is possible to select a low resolution image that has a pixel position close to the interpolation pixel position. Therefore, the high-resolution image generated based on the low-resolution image selected by the super-resolution processing can be a high-resolution image close to the image of the actual subject, and the highly-reliable high-resolution image. Can do.

  Embodiments of the present invention will be described with reference to the drawings. In the following description, an image pickup apparatus such as a digital camera or a digital video provided with an image processing apparatus (hereinafter referred to as “image processing unit”) that performs the image processing method according to the present invention will be described as an example. Further, as will be described later, a display device that performs digital processing of an image, such as a liquid crystal display or a plasma television, may be used as long as it has a similar image processing device.

(Configuration of imaging device)
First, the internal configuration of the imaging apparatus will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an internal configuration of the imaging apparatus.

  The imaging apparatus of FIG. 1 receives a solid-state imaging device (image sensor) 1 such as a CCD or CMOS sensor that converts light incident from a subject into an electrical signal, and an image signal that is an analog signal output from the image sensor 1. AFE (Analog FrontEnd) 2 for converting to a digital signal, a microphone 3 for converting sound input from the outside into an electric signal, and various image processing including super-resolution processing for an image signal to be a digital signal from the AFE 2 The image processing unit 4 that performs the processing, the audio processing unit 5 that converts the audio signal that is an analog signal from the microphone 3 into a digital signal, the image signal from the image processing unit 4, and the audio signal from the audio processing unit 5 A compression processing unit 6 that performs compression encoding processing such as MPEG (Moving Picture Experts Group) compression method, and the compression encoded signal compressed by the compression processing unit 6 in an external memory 15, the driver unit 7 to be recorded, the decompression processing unit 8 that decompresses and decodes the compressed encoded signal read from the external memory 20 by the driver unit 7, and the image by the image signal obtained by decoding by the decompression processing unit 8 A display unit 9 that displays the sound, an audio output circuit unit 10 that converts the audio signal from the decompression processing unit 8 into an analog signal, and a speaker unit 11 that reproduces and outputs audio based on the audio signal from the audio output circuit unit 10 A timing generator 12 that outputs a timing control signal for matching the operation timing of each block, a CPU (Central Processing Unit) 13 that controls the drive operation of the entire imaging apparatus, and each program for each operation A memory 14 for storing data and temporarily storing data during program execution, an operation unit 15 for inputting an instruction from a user, and a CP Comprising 13 and bus line 16 for exchanging data with each block, a memory 14 and a bus line 17 for exchanging data with each block.

  In this imaging apparatus, when the operation unit 15 instructs to perform an imaging operation, an image signal that is an analog signal obtained by the photoelectric conversion operation of the image sensor 1 is output to the AFE 2. At this time, the image sensor 1 is supplied with a timing control signal from the timing generator 12 to perform horizontal scanning and vertical scanning, and output an image signal as data for each pixel. In the AFE 2, when an image signal that is an analog signal is converted into a digital signal and input to the image processing unit 4, various image processing such as signal conversion processing for generating a luminance signal and a color difference signal is performed. .

  Further, in the image processing unit 4, when the operation unit 15 is required to increase the resolution of the image signal obtained from the image sensor 1 by digital zoom, super-resolution based on the image signals for a plurality of frames from the image sensor 1. Processing is performed. Further, a luminance signal and a color difference signal are generated based on the image signal that has been subjected to the super-resolution processing. Further, since the super-resolution processing is performed, as described later, the motion amount of the image signal of a plurality of frames is calculated, and the alignment of each frame is performed according to the motion amount.

  Then, the image signal subjected to the image processing by the image processing unit 4 is given to the compression processing unit 6. At this time, an audio signal which is an analog signal obtained by inputting the sound into the microphone 3 is converted into a digital signal by the audio processing unit 5 and given to the compression processing unit 6. As a result, the compression processing unit 6 compresses and encodes the image signal and the audio signal, which are digital signals, based on the MPEG compression encoding method, gives the image signal and the audio signal to the driver unit 7, and records them in the external memory 20. At this time, the compressed signal recorded in the external memory 20 is read out by the driver unit 7 and applied to the expansion processing unit 8 to be subjected to expansion processing to obtain an image signal. This image signal is given to the display unit 9 to display a subject image that is currently photographed through the image sensor 1.

  In the above description, the operation at the time of moving image shooting has been described. However, even when a still image shooting is instructed, the audio signal is not acquired by the microphone 3 and a compressed signal of only the image signal is recorded in the external memory 20. The basic operation is the same as that during moving image shooting. In the case of this still image shooting, not only the compressed signal for the still image shot by the operation unit 15 is recorded in the external memory 20, but also the compressed signal for the current image shot by the image sensor 1 is also stored in the external memory. 20 is temporarily recorded. As a result, the compression signal for the currently captured image is expanded by the expansion processing unit 8 so that the current image captured by the image sensor 1 is displayed on the display unit 9 and can be confirmed by the user. .

  When performing the imaging operation in this way, the timing generator 12 gives a timing control signal to the AFE 2, the video processing unit 4, the audio processing unit 5, the compression processing unit 6, and the expansion processing unit 8, and the image sensor 1 An operation synchronized with the imaging operation for each frame is performed. Further, when taking a still image, a timing control signal is given from the timing generator 12 to the image sensor 1, the AFE 2, the video processing unit 4, and the compression processing unit 6 based on the shutter operation by the operation unit 15. The operation timing of each unit is synchronized.

  When an instruction to reproduce a moving image or an image recorded in the external memory 20 is given through the operation unit 15, the compressed signal recorded in the external memory 20 is read out by the driver unit 7 and is decompressed by the decompression processing unit 8. Given to. Then, the decompression processing unit 8 decompresses and decodes the image signal and the audio signal based on the MPEG compression encoding method. Then, the image signal is given to the display unit 9 to reproduce the image, and the audio signal is given to the speaker unit 11 via the audio output circuit unit 10 to reproduce the audio. Thereby, the moving image based on the compressed signal recorded in the external memory 20 is reproduced together with the sound. Further, when the compressed signal is composed of only the image signal, only the image is reproduced on the display unit 9.

(Basic concept of super-resolution processing)
Next, the basic concept of super-resolution processing executed in the image processing unit 4 in the above-described imaging apparatus will be briefly described. In describing the basic concept of super-resolution processing, in order to simplify the description, image data is based on pixel values of a plurality of pixels lined up in a one-dimensional direction, and 2 taken at different times. It is assumed that super-resolution processing is performed using frame image data. The pixel value in the following description indicates a luminance value. FIG. 2A shows the luminance distribution of the subject imaged by the image sensor 1, and FIGS. 2B to 2D show images when the subject is imaged by the image sensor 1 at different times. Data (low-resolution actual image) is shown. FIG. 3 is a flowchart for generating image data that becomes a high-resolution image from the low-resolution real image in FIG. Further, FIG. 3 schematically shows signal transition in each flow.

  For the subject with the luminance distribution shown in FIG. 2A, the sample points of the first frame when imaged by the image sensor 1 at time T1 are S1, S1 + ΔS, S1 + 2ΔS, and at time T2 (T1 ≠ T2). Assume that the sample points of the second frame when imaged by the image sensor 1 are S2, S2 + ΔS, and S2 + 2ΔS. Further, at this time, it is assumed that the sample point S1 of the first frame and the sample point S2 of the second frame are misaligned due to camera shake or the like.

  Sample points S1, S1 + ΔS, and S1 + 2ΔS indicate sample points on the subject. In the low-resolution real image Fa that is the first frame shown in FIG. 2B, the sample points S1, S1 + ΔS, and S1 + 2ΔS are captured. The luminance values are the pixel values pa1, pa2, pa3 in the pixels P1, P2, P3. Sample points S2, S2 + ΔS, and S2 + 2ΔS indicate sample points on the subject. In the low-resolution real image Fb that is the second frame shown in FIG. 2C, the sample points S2, S2 + ΔS, and S2 + 2ΔS are captured. The luminance values are the pixel values pb1, pb2, and pb3 in the pixels P1, P2, and P3.

  Accordingly, the pixel values pa1, pa2, and pa3 of the pixels P1, P2, and P3 of the low-resolution real image Fa that becomes the first frame have the relationship as shown in FIG. 2B, and the low-resolution real image that becomes the second frame. The pixel values pb1, pb2, and pb3 in the pixels P1, P2, and P3 of the image Fb have a relationship as shown in FIG. As described above, the low-resolution real image Fa shown in FIG. 2B and the low-resolution real image Fb shown in FIG. 2C each have a pixel position of only (S1-S2) based on the position of the subject in the drawing. The image is shifted. When the low-resolution real image Fb is represented with reference to the pixels P1, P2, and P3 of the low-resolution real image Fa (that is, the low-resolution real image Fb is the amount of motion (S1-S2) relative to the low-resolution real image Fa). When the positional deviation is corrected), the low-resolution real image Fb is represented as shown in FIG.

  Then, as shown in FIG. 3A, the high-resolution image Fx1 is estimated by combining the low-resolution real images Fa and Fb as shown in FIG. 2B and FIG. (STEP 31). At this time, in order to simplify the following description, for example, it is assumed that the resolution is doubled in the one-dimensional direction. That is, as pixels of the high-resolution image Fx1, in addition to the pixels P1, P2, and P3 of the low-resolution real images Fa and Fb, a pixel P4 that is located at an intermediate position between the pixels P1 and P2, and an intermediate position between the pixels P2 and P3 It is assumed that the pixel P5 located at is set.

  If the low-resolution actual image Fa is the reference frame, the pixel values at the pixels P1, P2, and P3 are set to the pixel values pa1, pa2, and pa3 in the low-resolution actual image Fa. For the pixel P4, the distance of the pixel position (center position of the pixel) from the pixel P4 is greater than the distance from the pixel positions of the pixels P1 and P2 in the low-resolution real image Fa to the pixel P1 in the low-resolution real image Fb. Since the distance from the pixel position is closer, the pixel value pb1 is assumed. Similarly, for the pixel P5, the distance between the pixel position (pixel center position) and the pixel P5 is smaller than the distance from the pixel positions of the pixels P2 and P3 in the low-resolution real image Fa, and the pixel P2 in the low-resolution real image Fb. It is assumed that the pixel value pb2 is obtained because the distance from the pixel position is shorter. As described above, a high resolution image obtained by setting the pixel values of the pixels P1 to P5 as pa1, pa2, pa3, pb1, and pb2 may be estimated as the high resolution image Fx1.

  Thereafter, the high-resolution image Fx1 obtained in STEP 31 is subjected to calculation using a conversion equation having parameters such as a down-sampling amount, a blur amount, and a positional shift amount (corresponding to a motion amount) as shown in FIG. ), The low-resolution estimated images Fa1 and Fb1 that are the estimated images for the low-resolution actual images Fa and Fb are generated (STEP 32). In FIG. 3B, low-resolution estimated images estimated by the n-th processing, that is, low-resolution estimated images Fan and Fbn corresponding to the low-resolution actual images Fa and Fb estimated from the high-resolution image Fxn are shown. Show.

  That is, based on the high-resolution image Fx1, the pixel values at the sample points S1, S1 + ΔS, S1 + 2ΔS are estimated, and the low-resolution estimated image Fa1 having the acquired pixel values pa11 to pa31 as the pixel values of the pixels P1 to P3 is generated. . Similarly, based on the high-resolution image Fx1, the pixel values at the sample points S2, S2 + ΔS, S2 + 2ΔS are estimated, and the low-resolution estimated image Fb1 that uses the acquired pixel values pb11 to pb31 as the pixel values of the pixels P1 to P3 is generated. To do.

  Then, as shown in FIG. 3C, the difference between the low resolution estimated images Fa1 and Fb1 thus obtained and the low resolution actual images Fa and Fb is obtained, and the differences are synthesized. Thus, the difference image ΔFx1 with respect to the high resolution image Fx1 is acquired (STEP 33). In FIG. 3C, the difference image ΔFxn with respect to the high-resolution image Fxn acquired by the n-th process, that is, the difference images ΔFan and ΔFbn between the low-resolution estimated images Fan and Fbn and the low-resolution actual images Fa and Fb. A difference image ΔFxn obtained by combining the images is shown.

  By the process of FIG. 3C, the difference (pa11−pa1), (pa21−pa2), and (pa31−pa3) between the pixels P1, P2, and P3 in the low resolution estimated image Fa1 and the low resolution actual image Fa. A difference image ΔFb1 is obtained from the differences (pb11−pb1), (pb21−pb2), and (pb31−pb3) at the pixels P1, P2, and P3 in the image ΔFa1, the low resolution estimated image Fa2, and the low resolution actual image Fb. That is, in the difference image ΔFa1, the difference values (pa11−pa1), (pa21−pa2), and (pa31−pa3) are pixel values, and in the difference image ΔFb1, the difference values (pb11−pb1), (pb21−pb2), (Pb31-pb3) is the pixel value.

  Then, by combining the pixel values of the difference images ΔFa1 and ΔFb1, the difference values in the pixels F1 to F5 are calculated, and the difference image ΔFx1 with respect to the high resolution image Fx1 is acquired. When obtaining the difference image ΔFx1 by combining the pixel values of the difference images ΔFa1 and ΔFb1, for example, in the ML method or the MAP method, a square error is used as an evaluation function. That is, the evaluation function of the ML method or the MAP method becomes a value obtained by squaring the pixel values of the difference images ΔFa1 and ΔFb1 and adding them between frames. Therefore, the gradient, which is the differential value of the evaluation function, is a value obtained by doubling the pixel values of the difference images ΔFa1 and ΔFb1, so that the difference image ΔFx1 with respect to the high resolution image Fx1 has the pixel values of the difference images ΔFa1 and ΔFb1. It is calculated by increasing the resolution using the doubled value.

  Thereafter, as shown in FIG. 3D, the pixel values (difference values) of the pixels P1 to P5 in the obtained difference image ΔFx1 are obtained from the pixel values of the pixels P1 to P5 in the high resolution image Fx1 estimated in STEP1. By subtracting, the high-resolution image Fx2 having a pixel value close to the subject having the luminance distribution shown in FIG. 2A is reconstructed (STEP 34). In FIG. 3D, the high-resolution image Fx (n + 1) obtained by the n-th processing, that is, the high-resolution image Fx (n + 1) obtained by subtracting the difference image ΔFxn from the high-resolution estimated image Fxn. Indicates.

  Then, by repeating the processing of STEP 32 to STEP 34 described above, the pixel value of the difference image ΔFxn obtained in STEP 33 is reduced, and the pixel value of the high-resolution image Fxn is close to the subject having the luminance distribution shown in FIG. Converges to pixel values. In STEP 32 and STEP 34 in the n-th process, the low-resolution estimated images Fan and Fbn and the high-resolution estimated image Fx (n + 1) are obtained by the high-resolution image Fxn obtained in STEP 34 in the previous (n−1) -th process. To be acquired. Then, when the pixel value (difference value) of the difference image ΔFxn becomes smaller than a predetermined value, or when the pixel value (difference value) of the difference image ΔFxn is converged, in the previous process (n-1th process). The super-resolution processing is terminated with the high-resolution image Fxn obtained in STEP 34 as the target high-resolution image.

  As the super-resolution processing based on the above-described flow, the ML method, the MAP method, the POCS method, the IBP method, and the like mentioned in [Background Art] are used in the image processing unit 4. For example, when the MAP method is used, the difference image ΔFxn with respect to the high-resolution image Fxn acquired in STEP 33 is configured by a gradient image having a pixel value as a value obtained by differentiating the evaluation function in the MAP method by the gradient method. To do. That is, the high-resolution image Fxn constrained by the posterior probability distribution is added to the differential image of the high-resolution image Fxn obtained by combining the differential images ΔFan and ΔFbn obtained by the least square method using the square error at each pixel position as the pixel value. After adding the high-speed image, the gradient image corresponding to the difference image ΔFxn in STEP 33 is acquired by performing differentiation using the gradient method. In STEP 34, the high-resolution image Fx (n + 1) is reconstructed by subtracting the difference image ΔFxn based on the gradient image from the high-resolution image Fxn.

  When such super-resolution processing is performed, as described above, first, as shown in STEP 31, the high-resolution image is estimated from a plurality of frames of low-resolution real images photographed by the image sensor 1, so that the optimum It becomes possible to acquire a high resolution image. Then, it is necessary to select a low-resolution real image that is optimal when estimating the high-resolution image in STEP 31 from a plurality of low-resolution real images including the frame of interest. By selecting this optimum low-resolution real image, the acquisition speed of the high-resolution image can be increased, and the number of frames is limited, thereby reducing the load on the image processing unit 4 related to the calculation in the super-resolution processing. be able to. Further, a motion amount that is a positional shift amount generated between each of the selected low-resolution actual images of a plurality of frames is detected, and correction for the detected motion amount is performed. This positional deviation correction will be described later.

(Selection of low-resolution real image)
Hereinafter, a basic concept of selecting a low-resolution real image that is optimal in the above-described super-resolution processing will be described. In describing the basic concept of selecting a low-resolution real image, in order to simplify the description, image data is based on pixel values of a plurality of pixels arranged in a one-dimensional direction, and has different imaging times. It is assumed that a low resolution real image of 3 frames is selected from the low resolution real image of the frame. FIG. 4 shows a luminance distribution of a subject imaged by the image sensor 1 and a low-resolution real image when the subject is imaged by the image sensor 1 at different times. FIGS. 5 and 6 show the results of super-resolution processing using the selected three-frame low-resolution real images.

  As shown in FIG. 4A, the sample points of the first frame when imaged by the image sensor 1 at time T1 are S1 and S1 + ΔS, and imaged by the image sensor 1 at time T2 (T1 ≠ T2). The sample points of the second frame at the time are S2, S2 + ΔS, the sample points of the third frame when imaged by the image sensor 1 at time T3 (T3 ≠ T2, T3 ≠ T1) are S3, S3 + ΔS, and time Assume that the sample points of the fourth frame when imaged by the image sensor 1 at T4 (T4 ≠ T3, T4 ≠ T2, T4 ≠ T1) are S4 and S4 + ΔS. At this time, it is assumed that the sample points S1 to S4 of the first to fourth frames are caused to shift due to camera shake or the like.

  Thereby, the pixel values pa1 and pa2 in the pixels P1 and P2 of the low-resolution real image Fa that becomes the first frame have the relationship as shown in FIG. 4B, and the pixels of the low-resolution real image Fb that becomes the second frame. The pixel values pb1 and pb2 in P1 and P2 have the relationship shown in FIG. 4C, and the pixel values pc1 and pc2 in the pixels P1 and P2 of the low-resolution real image Fc that is the third frame are as shown in FIG. Further, the pixel values pd1 and pd2 in the pixels P1 and P2 of the low-resolution actual image Fd that is the fourth frame are as shown in FIG.

  Each of the low resolution real images Fa to Fd shown in FIGS. 4B to 4E is based on the pixels P1 and P2 corresponding to the sample points S1 to S4 and S1 + ΔS to S4 + ΔS of the low resolution real images Fa to Fd. Yes. In addition, as described above, when four frames of low resolution real images Fa to Fd as shown in FIGS. 4B to 4E are acquired at different imaging times, the low resolution real image Fa is used as a frame of interest. Assume that image processing is performed.

(1) When the resolution is doubled When a high-resolution image having a resolution twice that of the low-resolution real image is acquired in a one-dimensional direction, a position that is an intermediate point between the pixel positions of the low-resolution real images Fa to Fd (Intermediate position) is additionally set as the pixel position. That is, as shown in FIG. 5B, in the high-resolution image Fx, in addition to the pixels P1 and P2, an intermediate between pixels in the low-resolution real image is further obtained, such as a pixel Px at an intermediate point between the pixels P1 and P2. Pixels to be dots are added as interpolation pixels, and the high-resolution image Fx is an image having twice the number of pixels. That is, the interpolation pixel is a pixel that is interpolated between the pixels of the low resolution image when generating the high resolution image, and corresponds to the pixel Px of the high resolution image Fx in the example of FIG.

  When the high-resolution image Fx shown in FIG. 5B is acquired by super-resolution processing, first, as shown in FIG. 5A, the low-resolution real image is based on the pixels P1 and P2 of the low-resolution real image Fa. The positional deviations in Fa to Fd are corrected, and the low-resolution actual images Fa to Fd are aligned (corresponding to “positional deviation correction”). When the positional deviation based on the amount of motion is corrected as shown in FIG. 5A, the positions of the pixels P1 and P2 of the low-resolution real images Fb to Fd with respect to the pixels P1 and P2 of the low-resolution real image Fa are as shown in FIG. The amount of movement (S2-S1), (S3-S1), (S4-S1) between the sampling points S2 to S4 shown in (c) to FIG. 4 (e) and the sampling point S1 shown in FIG. ).

  That is, as shown in FIG. 5A, when the low resolution real images Fa to Fd are combined, the pixel values of the pixels P1 and P2 are the pixel values pa1 and pa2 of the pixels P1 and P2 of the low resolution real image Fa. Become. The pixel values PB1 and PB2 whose pixel positions are shifted from the pixels P1 and P2 by S2-S1 are the pixel values pb1 and pb2 of the pixels P1 and P2 of the low-resolution real image Fb (see FIG. 4C). It becomes. Similarly, pixel values PC1 and PC2 whose pixel positions are shifted from the pixels P1 and P2 by S3-S1 are pixel values pc1 and pc2 of the pixels P1 and P2 of the low-resolution real image Fc (see FIG. 4D). The pixel values of the pixel positions PD1 and PD2 whose pixel positions are shifted from the pixels P1 and P2 by S4-S1 are the pixel values pd1 and pd2 of the pixels P1 and P2 of the low-resolution real image Fd (FIG. 4 (e)). ))).

  When the positional deviation based on the amount of motion between each of the low-resolution actual images Fb to Fd and the low-resolution actual image Fa is corrected, the super-relationship between the pixels P1 and P2 of the low-resolution actual image Fa serving as the frame of interest is Two frames of low-resolution real images that are optimal for resolution processing are selected from the three frames of low-resolution real images Fb to Fd. Of the three frames of the low resolution real images Fb to Fd, the sample point is located at a position close to the intermediate position between the pixels of the low resolution real image Fa (corresponding to the pixel position of the pixel Px in the case of FIG. 5A). A low-resolution real image including the pixels to be selected is selected.

  That is, as shown in FIG. 5A, the amount of motion (S2-S1), (S3-S1), (S4-S1) between each of the low-resolution actual images Fb to Fd and the low-resolution actual image Fa is between the pixels. When the relationship with the distance ΔS is | ΔS / 2− | S2-S1 ||> | ΔS / 2− | S4-S1 ||> | ΔS / 2− | S3-S1 || The pixel positions of the sample points Fb to Fd are closer to the intermediate positions between the pixels of the low-resolution real image Fa in the order of Fc, Fd, and Fb. In FIG. 5A, the movement amounts (S2-S1), (S3-S1), (S4-S1) between the low-resolution actual images Fb to Fd and the low-resolution actual image Fa and the intermediate positions are shown. The magnitude relationship with the distance ΔS / 2 is (S2−S1) <(S3−S1) <ΔS / 2 <(S4−S1).

  Thereby, based on the relationship between the amount of motion between each of the low resolution actual images Fb to Fd and the low resolution actual image Fa, each of the low resolution actual images Fb to Fd is subjected to super-resolution processing in the order of Fc, Fd, and Fb. Judged to be suitable. Therefore, since three frames of low-resolution real images including the low-resolution real image Fa serving as the target frame are selected, the low-resolution real images Fa, Fc, and Fd are selected. Then, the above-described super-resolution processing is performed based on the selected low-resolution real images Fa, Fc, and Fd, whereby a high-resolution image Fx as shown in FIG. 5B is obtained.

  In the above description, in order to simplify the description, it is assumed that the low-resolution actual image is selected according to the relationship after the positional deviation correction (agreement with “alignment”) of the low-resolution actual images Fa to Fd. However, based on the amount of motion obtained from each sample point of the low resolution real images Fa to Fd, the intermediate position and distance between each pixel of the low resolution real image Fa serving as the frame of interest is calculated, and the calculated distance is short. It may be selected from low-resolution real images.

  As described above, when the resolution is doubled, when the positional deviation correction of a plurality of low resolution images is performed on the basis of the low resolution image serving as the frame of interest, an intermediate between pixels in the low resolution image serving as the frame of interest. Super-resolution processing is performed by selecting a low-resolution image including pixels at a position, that is, a position close to the interpolation pixel position. Thereby, since the pixel value of the real image at the pixel position close to the interpolation pixel position when acquiring the high resolution image can be acquired, the pixel value of the interpolation pixel is acquired based on the value close to the pixel value of the interpolation pixel. It can be set to a value close to the luminance value of the subject.

  Further, when the resolution is doubled, in the above example, the low-resolution real image of 3 frames including the target frame is selected from the low-resolution real image of 4 frames, but n (n is 3 or more). If a low-resolution real image of m (m is an integer of 2 to n-1) frames including the target frame is selected from the low-resolution real image of frames, the number of frames is not limited. Absent. In order to reduce the load on the arithmetic processing in the image processing unit 4, the number of frames of the low-resolution actual image of n frames to be selected and the low-resolution actual image of m frames to be selected is reduced. Is desirable.

(2) When the resolution is three times When acquiring a high resolution image having a resolution three times that of the low resolution real image in one dimension, the pixel positions of the low resolution real images Fa to Fd are divided into three equal parts. This position is additionally set as a pixel position. That is, as shown in FIG. 6B, in the high resolution images Fy and Fz, in addition to the pixels P1 and P2, the pixels P1 and Px2 are further divided into three equal parts, in addition to the pixels P1 and P2. Pixels that are positions that divide the pixels in the real image into three equal parts are added as interpolation pixels, and the high-resolution image Fy is made an image with three times the number of pixels.

  When the high resolution image Fy shown in FIG. 6B is acquired by super-resolution processing, as in the case where the resolution is doubled, first, as shown in FIG. 6A, the pixel P1 of the low resolution real image Fa is obtained. , P2 as a reference, the positional deviation based on the motion amount in the low-resolution real images Fa to Fd is corrected, and the low-resolution real images Fa to Fd are aligned. This alignment (correction of positional deviation) is the same as that in FIG. 5A when the resolution is doubled, and the positional deviation amount (motion amount) of the target frame with the low-resolution actual image Fa is the same value. It becomes.

  As shown in FIG. 6A, when the positional deviation between each of the low-resolution real images Fb to Fd and the low-resolution real image Fa is corrected, the positions of the pixels P1 and P2 of the low-resolution real image Fa serving as the frame of interest. Depending on the relationship, the low-resolution real image that is optimal as the super-resolution processing is selected from the three-frame low-resolution real images Fb to Fd. At this time, the pixel position serving as a reference for selection differs depending on the number of frames to be selected. That is, in the example in which the resolution is doubled, the intermediate position between the pixels in the low-resolution actual image and the interpolation pixel position are the same position. However, when the resolution is tripled, the intermediate position between the pixels and the interpolated pixel position in the low-resolution real image are different. Therefore, (1-a) a case where the intermediate position between each pixel of the low-resolution real image Fa serving as the target frame is used as a reference for determination, and (1-b) an interpolation pixel position in the high-resolution image is used as a reference for determination. The case is determined by the number of frames of the low-resolution real image to be selected.

(1-a) Example based on an intermediate position between each pixel of the target frame The reference for frame selection in this example is, for example, a low resolution of another frame other than the low-resolution real image Fa that is the target frame. The number of frames to be selected as the low-resolution real image that is optimal as the super-resolution processing is arranged between the pixels in the low-resolution real image so that the real image is selected from the three-frame low-resolution real images Fb to Fd. Less than the number of interpolation pixels (when the resolution is tripled as in this example, the interpolation pixels are interpolated by two pixels between pixels in the low-resolution real image, so the number of interpolation pixels is 2). It is sometimes preferred that it is mainly used.

  As described above, when selecting the optimum low resolution image in the super-resolution processing with reference to the intermediate position between the pixels of the low resolution real image Fa as the target frame, as in the case of doubling the resolution, The low-resolution real image whose sample point is close to the intermediate position between the pixels of the low-resolution real image Fa is selected. That is, for example, when one frame is selected from the three low-resolution real images Fb to Fd in addition to the low-resolution real image Fa serving as the target frame, as described above, depending on the sample points of the low-resolution real images Fb to Fd. Since the pixel position is close to the intermediate position between the pixels of the low-resolution real image Fa in the order of Fc, Fd, and Fb (see FIGS. 5A and 6A), the low-resolution real image Fc is Selected. Therefore, the super-resolution processing described above is performed using the low-resolution real images Fa and Fc, and a high-resolution image Fy as shown in FIG. 6B is obtained.

  Note that when setting the standard as in this example, even if the number of frames to be selected for a low-resolution real image that is optimal for super-resolution processing is large, it is effective for reducing the processing load on the selection operation. It is. Therefore, in the case where two frames are selected from the three low-resolution real images Fb to Fd in addition to the low-resolution real image Fa serving as the target frame, as described above, the pixel positions based on the sample points of the low-resolution real images Fb to Fd However, since the intermediate positions between the pixels of the low-resolution real image Fa are close to the order of Fc, Fd, and Fb (see FIGS. 5A and 6A), the low-resolution real images Fc and Fd are Selected. Therefore, the above-described super-resolution processing is performed on the low-resolution real images Fa, Fc, and Fd, and a high-resolution image Fz as shown in FIG. 6C is obtained.

(1-b) An example in which the interpolation pixel position in the high-resolution image is used as a criterion for determination In this example, the criterion for frame selection is, for example, a low resolution of two frames other than the low-resolution real image Fa serving as the frame of interest. The number of frames to be selected as the low-resolution real image that is optimal as the super-resolution processing is arranged between the pixels in the low-resolution real image so that the real image is selected from the three-frame low-resolution real images Fb to Fd. The number of interpolated pixels (in this example, when the resolution is tripled, the interpolated pixels are interpolated by two pixels between pixels in the low-resolution real image, so the number of interpolated pixels is 2). It is preferably used when

  As described above, when an optimal low resolution image is selected in the super-resolution processing based on the interpolation pixel position of the generated high resolution image, the sample point is 1/3 between each pixel of the low resolution real image Fa. Alternatively, the near-resolution real image is selected at a position that is 2/3. That is, a position where the distance from each pixel of the low-resolution real image Fa is ΔS / 3 or 2ΔS / 3 (corresponding to the pixel positions of the interpolation pixels Px1 and Px2 in the high-resolution images Fy and Fz in FIG. 6B). Are selected in order from a low-resolution image whose sample points are close to the reference.

  That is, as shown in FIG. 6A, the motion amounts (S2-S1), (S3-S1), (S4-S1) between the low-resolution actual images Fb to Fd and the low-resolution actual image Fa are between the pixels. When the relationship with the distance ΔS is | ΔS / 3− | S2−S1 || = | ΔS / 3− | S3−S1 ||> | 2ΔS / 3− | S4−S1 || The pixel position by the sample points of Fb to Fd is closest to the sample point of the low-resolution real image Fd with respect to the position where the distance from each pixel of the low-resolution real image Fa is ΔS / 3 or 2ΔS / 3 Become. Then, the sample points of the low-resolution actual images Fb and Fc are equal to the position where the distance from each pixel of the low-resolution actual image Fa is ΔS / 3.

  In FIG. 6A, the motion amounts (S2-S1), (S3-S1), (S4-S1) and the interpolation pixels between the low-resolution actual images Fb to Fd and the low-resolution actual image Fa are shown. The magnitude relationship between the distances ΔS / 3 and 2ΔS / 3 is (S2−S1) <ΔS / 3 <(S3−S1) <(S4−S1) <2ΔS / 3. The magnitude relationship between the amount of motion (S2-S1) and (S3-S1) between the low-resolution actual images Fb to Fd and the low-resolution actual image Fa and the distance ΔS / 2 to the intermediate position is (S2-S1). ) <(S3-S1) <ΔS / 2. That is, the low resolution images Fb and Fc are close to the order of Fc and Fb with respect to the intermediate position between the pixels of the low resolution actual image Fa.

  Thus, based on the relationship between the amount of motion between each of the low-resolution actual images Fb to Fd and the low-resolution actual image Fa, first, the low-resolution actual image Fd having the position closest to the interpolation pixel position in the high-resolution image as a sample point. Is determined to be suitable for super-resolution processing. Since the low-resolution actual image whose sample point is next to the interpolation pixel position is two frames of the low-resolution actual images Fb and Fc, it is necessary to select one frame from these two frames. At this time, the low-resolution real image Fc whose sample point is close to the intermediate position between the pixels of the low-resolution real image Fa is selected. In this way, the low-resolution real images Fc and Fd are selected, and the above-described super-resolution processing is performed using the low-resolution real images Fa, Fc, and Fd, and a high resolution as shown in FIG. An image Fz is obtained.

  A case different from the case of FIG. 6A will be described below with reference to FIG. FIG. 7 is a diagram illustrating the positional relationship between the sample points S1 to S4 of the low-resolution real images Fa to Fd. For example, as shown in FIG. 7A, the motion amounts (S2-S1), (S3-S1), and (S4-S1) between the low-resolution actual images Fb to Fd and the low-resolution actual image Fa are between the pixels. When the relationship with the distance ΔS is | ΔS / 3− | S2-S1 ||> | ΔS / 3− | S3-S1 ||> | 2ΔS / 3− | S4-S1 || And the distance between the interpolated pixels of the high resolution image. That is, in the case of FIG. 7A, since the distance between the sample point and the interpolation pixel of the high resolution image is close to Fd, Fc, and Fb in this order, the low resolution real images Fd and Fc are selected.

  In addition, as shown in FIG. 7B, when | ΔS / 3− | S3-S1 ||> | ΔS / 3− | S2−S1 |||| 2−ΔS / 3− | S4−S1 || Since the distance between the sample point and the interpolation pixel of the high resolution image is closer to Fd, Fb, and Fc in this order, the low resolution real images Fd and Fb are selected. 7A and 7B, the low-resolution actual images Fb to Fd, the low-resolution actual image Fa, and the motion amounts (S2-S1), (S3-S1), (S4-S1) The relationship between the distances to the interpolation pixels ΔS / 3 and 2ΔS / 3 is such that (S2−S1) <ΔS / 3 <(S3−S1) <ΔS / 2 <(S4−S1) <2ΔS / 3. And

  Further, as shown in FIGS. 7C and 7D, | ΔS / 3− | S2−S1 || = | 2ΔS / 3− | S3−S1 ||> | 2ΔS / 3− | S4−S1 When || becomes, first, the low-resolution real image Fd having the shortest distance between the sample point and the interpolation pixel of the high-resolution image is selected. Then, as shown in FIG. 7C, when ΔS / 3 <(S2−S1) <ΔS / 2 <(S3−S1) <(S4−S1) <2ΔS / 3, the low-resolution real image Fa Since the low-resolution real image Fd whose sample point is a position close to the interpolation pixel (corresponding to the interpolation pixel Px2) whose distance from the pixel is 2ΔS / 3 has already been selected, the distance from the pixel of the low-resolution real image Fa A low-resolution real image Fb having a sample point at a position close to an interpolation pixel (corresponding to the interpolation pixel Px1) having a ΔS / 3 is selected.

  Furthermore, as shown in FIG. 7D, when (S2-S1) <ΔS / 3 <ΔS / 2 <(S3-S1) <(S4-S1) <2ΔS / 3, As in the case, the low-resolution real image Fb having a sample point at a position close to an interpolation pixel (corresponding to the interpolation pixel Px1) whose distance from the pixel of the low-resolution real image Fa is ΔS / 3 may be selected. Alternatively, as in the case of FIG. 6A, the low-resolution real image Fc having a sample point at a position close to the intermediate position between the pixels of the low-resolution real image Fa may be selected.

  In the case of FIG. 7D, the reference for selecting which of the low-resolution actual images Fb and Fc is selected may be changed depending on the length of the distance to the interpolation pixel. That is, when the distance between the interpolation pixel and the sample point of the low-resolution actual images Fb and Fc is shorter than a predetermined value, the position near the interpolation pixel where the distance from the pixel of the low-resolution actual image Fa is ΔS / 3 is the sample point. A low resolution real image Fb is selected. On the other hand, when the distance between the interpolation pixel and the sample point of the low-resolution real images Fb and Fc is longer than a predetermined value, the low-resolution real image with the position close to the intermediate position between the pixels of the low-resolution real image Fa as the sample point. Select the image Fc.

  As described above, a case has been described in which a high-resolution image having a resolution three times that of the low-resolution actual image has been obtained. Can be selected. In other words, when the number of frames to be selected is small or when the calculation process in the selection operation is simplified, a position close to the intermediate position between each pixel of the low-resolution real image that is the frame of interest is used as a reference. A low-resolution real image for performing the resolution process is selected.

  When the number of frames to be selected is increased, the selection is first made based on the interpolation pixel position of the high resolution image. When there is a low-resolution real image in which the distance between the interpolation pixel position and the sample point is equal, the interpolation pixel position that is not the target for obtaining the distance from the sample point of the selected low-resolution real image, Based on the relationship between the intermediate position between each pixel of the low-resolution real image to be a frame and the distance between the sample points, the low-resolution real image for performing the super-resolution processing is selected.

(Position misalignment correction)
Hereinafter, the positional deviation correction operation will be briefly described. This misalignment correction is performed on the low-resolution real image, and is performed with reference to the low-resolution real image that is the frame of interest in the super-resolution processing. That is, the low-resolution actual image other than the target frame selected as the optimal low-resolution real image for the super-resolution processing is corrected for the positional deviation with respect to the low-resolution actual image serving as the reference target frame. At this time, after detecting the amount of motion for each of two consecutive low-resolution images, the amount of motion with the low-resolution actual image serving as the frame of interest is obtained by obtaining the sum of the detected amounts of motion. Then, based on the obtained amount of motion, position shift correction of the low-resolution image optimal for super-resolution processing other than the frame of interest is performed.

  That is, in detecting the amount of motion, first, one of two temporally continuous low-resolution images is detected as a reference image, and the other is detected as a non-reference image. In the case of a low-resolution image that is the n-th frame counted from the target frame, the amount of motion with the low-resolution image that is the target frame is detected by calculating the total amount of motion between the low-resolution images of n frames. . This detection of the amount of motion is also used when selecting a low-resolution real image optimal for super-resolution processing.

  In this motion amount detection, the motion amount between the reference image and the non-reference image, which are two consecutive low-resolution images, is detected. For this motion amount detection, for example, a well-known representative point is used. The amount of motion in pixels between images is detected by the matching method, and then the amount of motion in one pixel is detected based on the relationship between the pixel values of neighboring pixels that have been corrected for positional deviation in pixels. I do not care. Hereinafter, the representative point matching method and the motion amount detection within one pixel will be briefly described.

(1) Representative Point Matching Method For each of the reference image and the non-reference image, for example, as shown in FIG. 8, an a × b pixel group (for example, a 36 × 36 pixel group) is set as one small region e. Further, the small region e group (for example, 6 × 8 small region e group) of the p × q region is further divided as one detection region E. In the reference image, in each small area e, as shown in FIG. 9A, one pixel is set as the representative point R from the a × b pixels constituting the small area e. In the non-reference image, In each small region e, as shown in FIG. 9B, a plurality of pixels among a × b pixels constituting the small region e are set as sampling points S (for example, all the pixels of a × b May be used as the sampling point S).

  Thus, when the small area e and the detection area E are set, the pixel value of each sampling point S of the non-reference image and the representative point R of the reference image for the small area e at the same position of the reference image and the non-reference image. A difference from the pixel value is obtained as a correlation value at each sampling point S. Then, for each detection region E, the correlation values of the sampling points S whose relative positions to the representative point R are the same among the small regions e are cumulatively added for all the small regions e constituting the detection region E. Thus, the cumulative correlation value at each sampling point S is acquired. Thus, for each detection region E, the correlation values of p × q sampling points S having the same relative position to the representative point R are cumulatively added, so that cumulative correlation values for the number of sampling points are obtained. (For example, when all the pixels of a × b are set as the sampling point S, a × b cumulative correlation values are obtained).

  In this way, when the cumulative correlation value for each sampling point S is obtained for each detection region E, the cumulative correlation value considered to have the highest correlation with the representative point R in each detection region E is the minimum. A sampling point S is detected. In each detection region E, the amount of motion between the sampling point S and the representative point R that minimizes the cumulative correlation value is obtained from each pixel position. Thereafter, by averaging the motion amounts obtained for each detection region E, this average value is detected as a motion amount in pixel units between the reference image and the non-reference image.

(2) Motion amount detection within one pixel As described above, when a motion amount in pixel units is detected, a motion amount within one pixel is further detected. At this time, for example, for each small region e shown in FIG. 8, first, the correlation between the representative point R and the representative point R based on the pixel value of the pixel that becomes the representative point R of the reference image and the amount of motion obtained by the representative point matching method described above. The amount of motion in one pixel is detected based on the relationship between the pixel value of the high sampling point Sx and the pixel values of the surrounding pixels.

  That is, in each small region e, the pixel value La (see FIG. 10A) of the representative point R that becomes the pixel position (ar, br) in the reference image and the pixel position (as, bs) in the non-reference image. The pixel value Lb of the sample point Sx, the pixel value Lc (see FIG. 10B) of the pixel position (as + 1, bs) adjacent to the sample point Sx in the horizontal direction, and the pixel position adjacent to the sample point Sx in the vertical direction The amount of motion within one pixel is detected based on the relationship with the pixel value Ld (see FIG. 10B) of (as, bs + 1). At this time, according to the representative point matching method, the amount of movement of the pixel unit from the reference pixel to the non-reference image becomes a value represented by a vector amount of (as-ar, bs-br).

  Further, as shown in FIG. 11A, the pixel value Lb is changed linearly from the pixel value Lb by shifting one pixel in the horizontal direction from the pixel that becomes the sample point Sx, as shown in FIG. Furthermore, it is assumed that the pixel value Lb changes linearly from the pixel value Ld by shifting one pixel in the vertical direction from the pixel that is the sample point Sx. 10A and 10B, the horizontal position Δx (= (La−Lb) / (Lc−Lb) where the pixel value La is between the pixel values Lb and Lc. ) And a vertical position Δy (= (La−Lb) / (Ld−Lb)) that is the pixel value La between the pixel values Lb and Ld. That is, the vector amount represented by (Δx, Δy) is obtained as the amount of motion within one pixel between the reference pixel and the non-reference pixel.

  In this way, when the motion amount within one pixel in each of the small regions e is obtained, the obtained motion amount is averaged, and this average value is obtained as a motion in pixel units between the reference image and the non-reference image. Detect as quantity. The motion amount between the reference image and the non-reference image can be calculated by adding the motion amount in one pixel obtained to the motion amount in units of pixels obtained by the representative point matching method.

  When the amount of motion between two consecutive low-resolution images is detected by this amount-of-motion detection, the sum of the amounts of motion between the low-resolution images of a plurality of frames up to the low-resolution image serving as the frame of interest is calculated. Thus, the amount of motion with the low-resolution image that is the frame of interest is detected. The low-resolution image that is optimal for super-resolution processing can be selected based on the amount of motion acquired by detecting the amount of motion using the above method, and the low-resolution image that is the frame of interest for the selected low-resolution image It is possible to perform positional deviation correction based on the image. Note that the motion amount detection has been described by taking the above method as an example, but as long as the motion amount within a pixel can be detected, another method is used to detect a low-resolution image (reference frame) that becomes a frame of interest. The amount of motion between the image) and another low-resolution image (non-reference image) may be detected.

(Configuration example of image processing unit)
FIG. 12 shows a configuration example of the image processing unit 4 that performs super-resolution processing, low-resolution image selection processing, motion amount detection, and positional deviation correction processing as described above. The configuration of the image processing unit 4 shown in the block diagram of FIG. 12 will be briefly described below.

  The image processing unit 4 shown in FIG. 12 includes a frame memory 41 that temporarily stores an image signal that is a low-resolution real image for a plurality of frames converted into digital signals by the AFE 2, and a low resolution of the current frame that is input from the AFE 2. A motion amount calculation unit 42 that calculates an amount of motion given an image signal based on an actual image and an image signal based on a low-resolution actual image one frame before the current frame stored in the frame memory 41, and a motion amount calculation unit 42 Based on the detected amount of motion, an image evaluation unit 43 that selects an optimal low-resolution real image for performing super-resolution processing on the low-resolution real image of the current frame from the AFE 2, and the image evaluation unit 43 selects the image. A position shift correction unit 44 that receives the instructed low resolution actual image from the frame memory 41 and corrects the position shift of the current frame with the low resolution actual image; and AF The super-resolution calculation is performed by providing the low-resolution real image of the current frame from 2 and the low-resolution real image selected by the image evaluation unit 43 whose positional deviation has been corrected by the positional deviation correcting unit 44. Unit 45, and a signal processing unit 46 that generates a luminance signal and a color difference signal from an image signal based on a high-resolution image that has been subjected to super-resolution processing by the super-resolution calculation unit 45 or an image signal that is directly given from the AFE 2. Prepare.

  The operation of the image processing unit 4 configured as described above will be described. When high resolution by the operation unit 15 is not required, the image signal converted into a digital signal by the AFE 2 is given to the signal processing unit 46 for each frame. Then, the signal processing unit 46 generates a luminance signal and a color difference signal from the given image signal, and gives the acquired luminance signal and color difference signal to the compression processing unit 6 for each frame, whereby the compression processing unit 6 Is compressed and encoded.

  When a higher resolution is requested by the operation unit 15, the image signal converted into a digital signal by the AFE 2 is sent to the frame memory 41, the motion amount calculation unit 42, and the super-resolution calculation unit 45 for each frame. Given. Then, the image evaluation unit 43 sets a selection criterion as described above according to the designated resolution and the number of frames of the low-resolution real image to be selected. That is, a reference position for selecting a low-resolution real image suitable for super-resolution processing is determined from a position that is an intermediate point between pixel positions of the low-resolution real image and an interpolation pixel position of the high-resolution real image.

  When each block is set in this way, the low-resolution real image of the current frame given from the AFE 2 is given to the super-resolution computing unit 45, and is used as a frame of interest for performing super-resolution processing. Also, the motion amount calculation unit 42 obtains the motion amount between the low-resolution actual image of the current frame and the low-resolution actual image one frame before the current frame by the above-described motion amount detection process. Then, the amount of motion between the two consecutive low-resolution real images is given to the image evaluation unit 43 and stored therein.

  As described above, the image evaluation unit 43 adds the amount of motion between the low-resolution real images obtained every two consecutive frames, so that the low-resolution of a plurality of frames temporarily stored in the frame memory 41 is obtained. The amount of motion between each real image and the low-resolution real image of the current frame given by the AFE 2 is obtained. Then, the super-resolution processing is performed on the basis of the amount of motion between the low-resolution real image of the current frame, which is obtained for all the low-resolution real images of a plurality of frames temporarily stored in the frame memory 41. Select a suitable low-resolution real image. That is, when an N-frame low-resolution real image is used in the super-resolution calculation unit 45, the image evaluation unit 43 performs the above-described low-resolution real-image selection process to reduce the current frame that is the target frame. A low resolution actual image of N−1 frames is selected from the low resolution actual images stored in the frame memory 41 according to the amount of motion between the resolution actual images.

  In this way, when the low-resolution real image is selected by the image evaluation unit 43, the frame given to the positional deviation correction unit 44 is notified to the frame memory 41, and the amount of motion calculated for the selected frame Is notified to the positional deviation correction unit 44. Therefore, the frame memory 41 sends an image signal, which is a low-resolution real image of the frame notified from the image evaluation unit 43 to be selected for the super-resolution processing, to the positional deviation correction unit 44.

  Then, the frame memory 41 deletes an image signal that becomes a low-resolution real image of a frame that is not selected by the image evaluation unit 43. When an image signal of a frame that has not been selected is deleted from the frame memory 41, the image signal that is the current frame (corresponding to the frame of interest) input from the AFE 2 is excluded from the deletion target. And becomes a selection target in the next frame. As described above, since the image signal of the frame not selected is deleted, when the image signal of the next frame is input from the AFE 2, the image evaluation unit 43 remains stored in the frame memory 41 without being deleted. The image signal of the frame that is optimal for the super-resolution processing is selected from the image signals of the current frame.

  In addition, when an image signal that becomes a low-resolution real image of a frame selected for use in super-resolution processing is given to the positional deviation correction unit 44 for each frame, the current frame (corresponding to the frame of interest) with respect to the given frame. ) From the image evaluation unit 43 is also provided to the positional deviation correction unit 44. Then, the position shift correction unit 44 performs the above-described position shift correction operation based on the amount of motion from the image evaluation unit 43, and the pixel position is based on the pixel position of the low-resolution real image of the current frame as a reference. Changed to position. Then, the image signal of the low-resolution real image that has been subjected to the position shift correction by the position shift correction unit 44 is supplied to the super-resolution calculation unit 45.

  In the super-resolution calculation unit 45, the image signal that becomes the low-resolution real images of all the frames selected for the super-resolution processing by the image evaluation unit 43 is subjected to the positional shift correction by the positional shift correction unit 44 and given. At the same time, the AFE 2 gives an image signal that is a low-resolution real image of the current frame that is the frame of interest. Then, by applying the above-described super-resolution processing to the image signals given from the AFE 2 and the positional deviation correction unit 44, a high-resolution image that has been made high-resolution from the original low-resolution real image is obtained. . An image signal based on the high-resolution image obtained in this way is supplied to the signal processing unit 46. Then, the signal processing unit 46 generates a luminance signal and a color difference signal from the given image signal that is a high-resolution image of one frame, and then sends it to the compression processing unit 6.

  In this example, in the frame memory 41, the image signal that becomes the low-resolution real image of the frame not selected for the super-resolution processing is deleted, but the frame of the image signal input from the AFE 2 is updated. Each time, the image signal that becomes the oldest low-resolution real image stored for each frame may be deleted.

  The low resolution actual image of the current frame that has just been input to the AFE 2 is set as the frame of interest in the super resolution processing in the super resolution calculation unit 45. The image may be a frame of interest in the super-resolution processing in the super-resolution calculation unit 45. At this time, every time the frame of the image signal input from the AFE 2 is updated, the frame memory 41 deletes the image signal that becomes the low-resolution real image of the frame for which the super-resolution processing is completed.

(First example of super-resolution operation unit)
Further, a first example of the super-resolution calculation unit 45 in the image processing unit 4 will be described below by taking as an example a super-resolution calculation by the MAP method. FIG. 13 is a block diagram showing an internal configuration of the super-resolution calculation unit 45. In the configuration example of FIG. 13, it is assumed that the super-resolution calculation unit 45 generates a high-resolution image by selecting a low-resolution real image for two frames from the frame memory 41.

  The super-resolution calculation unit 45 shown in FIG. 13 is provided with a low-resolution real image that is the current frame from the AFE 2 and a low-resolution real image for two frames that have been subjected to position shift correction by the position shift correction unit 44. One of an initial high resolution estimation unit 451 that estimates a high resolution image to be a high resolution image, a high resolution image estimated by the initial high resolution estimation unit 451, and a high resolution image temporarily stored in the frame memory 455 A selection unit 452 that selects one to give to the subsequent stage, and a high-resolution update amount calculation unit that obtains an update amount of the high-resolution image from the high-resolution image selected by the selection unit 452 and the low-resolution real image for three frames 453, a subtraction unit 454 for subtracting the update amount obtained by the high resolution update amount calculation unit 453 from the high resolution image selected by the selection unit 452, and the high resolution image generated by the subtraction unit 454 It includes a frame memory 455 for storing, for a. The selection unit 452 selects the initial high resolution image estimated by the initial high resolution estimation unit 451 in the first selection operation, and the high resolution temporarily stored in the frame memory 455 in the second and subsequent selection operations. Select an image.

  Next, the operation of the super-resolution calculation unit 45 configured as described above will be described with reference to FIG. Note that u is the number of pixels of the low-resolution image including the low-resolution real image, and v is the number of pixels of the high-resolution image including the initial high-resolution image. Then, each pixel value of the low-resolution actual image Fa using u pixels is Ya = [ya1, ya2,..., Au], and each pixel value of the low-resolution actual image Fb using u pixels is Yb = [yb1, yb2,. ybu], and each pixel value of the low-resolution real image Fb with u pixels becomes Yc = [yc1, yc2,..., ycu]. When the super-resolution processing in the image processing unit 4 is performed, the low-resolution real image Fa that is the current frame is supplied from the AFE 2 to the initial high-resolution estimation unit 451. At this time, the low-resolution actual images Fb and Fc for two frames selected by the image evaluation unit 43 from the frame memory 41 and subjected to the position shift correction by the position shift correction unit 44 are also given to the initial high resolution estimation unit 451. It is done.

(1) Processing by Initial High-Resolution Image As a result, the initial high-resolution estimation unit 451 generates 3 frames based on the motion amount obtained by the motion amount calculation unit 42 as in STEP 31 shown in FIG. The positional deviation between the low-resolution real images Fa to Fc is detected. Then, an initial high-resolution image Fx1 with v pixels is obtained by performing an interpolation process using a positional shift confirmed by the amount of motion on the pixel values Ya to Yc of the low-resolution real images Fa to Fc. It is assumed that the pixel values of the high resolution images Fx1 to Fxn are X = [x1, x2,..., Xv]. When the initial high-resolution image Fx1 is generated by the initial high-resolution estimation unit 451, a filter that performs noise reduction processing using a median filter or the like is used as a part of the high-resolution image generation filter, It is preferable to increase the S / N ratio (signal / noise ratio) by increasing the number of taps of the filter.

  When the initial high resolution image Fx1 is output from the initial high resolution estimation unit 451 in this way, the initial high resolution image Fx1 is selected by the selection unit 452 and provided to the high resolution update amount calculation unit 453. In the high-resolution update amount calculation unit 453, the position of each of the low-resolution real images Fa to Fc with respect to the high-resolution real image Fx is given based on the amount of motion by being given the motion amount obtained by the motion amount calculation unit 42. Displacement and blur due to low resolution are calculated. Camera parameters including positional deviation and blur obtained from the amount of motion, and down-sampling amounts from v-pixel high-resolution image Fxn to low-resolution estimated images Fan-Fcn corresponding to u-pixel low-resolution actual images Fa-Fc Matrixes Wa to Wc are obtained.

Then, the initial high resolution image Fx1 given through the selection unit 452 is multiplied by the camera parameter matrix Wa to Wc, so that the low resolution real images Fa and Fb are obtained as in STEP32 shown in FIG. 3B. , Fc1 (= Wa · X), Fb1 (= Wb · X), and Fc1 (= Wc · X) are estimated. Further, the high-resolution update amount calculation unit 453 is provided with the low-resolution actual images Fa to Fc for three frames from the AFE 2 and the positional deviation correction unit 44, so that the estimated low-resolution estimated images Fa1 to Fc1 and the low resolution Differences | Wa · X−Ya | ² to | Wc · X−Yc | ² due to square errors with the actual images Fa to Fc are obtained.

The evaluation function I based on the difference between the low-resolution estimated images Fa1 to Fc1 and the low-resolution actual images Fa to Fc is added with a constraint term obtained by the high-resolution image selected by the selection unit 452 to the added value of each difference. It is required in the form. That is, the obtained evaluation function I is | Wc · X−Yc | ² + | Wb · X−Yb | ² + | Wc · X−Yc | ² + γ | CX | ² . In the constraint term γ | CX | ² of the evaluation function I, the matrix C is a matrix based on the prior probability model, and based on the prior knowledge that “the high-resolution image has few high-frequency components”, for example, Laplacian It is composed of a high-pass filter such as a filter. The coefficient γ is a parameter representing the strength of the weight in the evaluation function I of the constraint term.

  Since the evaluation function I thus obtained is minimized by the gradient method, the gradient ∂I / ∂X (= 2 × {WaT · (Wa · X−Ya) + WbT · (Wb (X−Yb) + WcT · (Wc · X−Yc) + γCT · C · X}). By obtaining the gradient ∂I / ∂X that is the update amount, the high-resolution update amount calculation unit 453 obtains the difference image ΔFx1 in STEP 33 shown in FIG. The subscript “T” represents a transposed matrix.

  When the update amount based on the gradient ∂I / ∂X calculated in this way is given from the high-resolution update amount calculation unit 453 to the subtraction unit 454, the subtraction unit 454 selects as shown in STEP 34 in FIG. The update amount by the gradient ∂I / ∂X is subtracted from the initial high resolution image Fx1 given from the initial high resolution estimation unit 451 via the unit 452. As a result of the subtraction by the subtraction unit 454, a high-resolution image Fx2 in which the pixel values of the v pixels are (X−∂I / ∂X) is obtained. Then, the newly obtained high-resolution image Fx2 is output from the subtracting unit 454, given to the frame memory 455, and temporarily stored.

(2) Processing by high-resolution image after generation As described above, the initial high-resolution image Fx1 is acquired, the update amount of the high-resolution image is gradient ∂I / ∂X, and a new high-resolution image Fx2 is acquired. Then, the selection unit 452 selects the high-resolution image stored in the frame memory 455 and gives it to the high-resolution update amount calculation unit 453. That is, in the case of n (n ≧ 2) arithmetic processing, the high-resolution image Fxn stored in the frame memory 455 is read by the selection unit 452 and provided to the high-resolution update amount calculation unit 453.

  Therefore, the high resolution update amount calculation unit 453 estimates the low resolution estimated images Fan (= Wa · X) to Fcn (= Wc · X) from the high resolution image Fxn stored in the frame memory 455. Based on the low-resolution estimated images Fan to Fcn thus obtained, the input low-resolution real images Fa to Fc, and the high-resolution image Fxn, 更新 I / ∂X (= 2 × {WaT The difference image ΔFxn by (Wa · X−Ya) + WbT · (Wb · X−Yb) + WcT · (Wc · X−Yc) + γCT · C · X}) is obtained by the high resolution update amount calculation unit 453. Then, in the subtraction unit 454, the difference image ΔFxn obtained by the high resolution update amount calculation unit 453 is subtracted from the high resolution image Fxn given from the frame memory 455 via the selection unit 452, that is, X−∂I When the calculation of / X is performed, a new high-resolution image Fx (n + 1) is obtained.

  In the super-resolution calculation unit 45, when the number of calculation processes by the high-resolution update amount calculation unit 453 is set to n, the high-resolution image Fx (n + 1) newly obtained by the subtraction unit 454 is The signal is output from the subtraction unit 454 to the signal processing unit 46. Further, in the super-resolution calculation unit 45, when the number of calculation processes by the high-resolution update amount calculation unit 453 is set to n + 1 or more, the high-resolution image Fx (n + 1) is given to the frame memory 455 and temporarily stored. Then, the (n + 1) th arithmetic processing is performed.

  Regarding the number of repetitions of arithmetic processing by the high-resolution update amount calculation unit 453 and the subtraction unit 454 in the super-resolution calculation unit 45, when the shooting interval of one frame is short, such as moving image shooting or continuous shooting of still images Is set so that the number of times decreases. In addition, in the case of normal still image shooting other than continuous shooting, the number of times may be set to be larger than in the case of moving image shooting or still image continuous shooting.

(Second example of super-resolution operation unit)
A second example of the super-resolution operation unit 45 in the image processing unit 4 described above will be described below by taking as an example a super-resolution operation by the MAP method. FIG. 14 is a block diagram showing an internal configuration of the super-resolution calculation unit 45. As shown in FIG.

  14 is replaced with the selection unit 452, the high-resolution update amount calculation unit 453, the subtraction unit 454, and the frame memory 455 in the configuration of the super-resolution calculation unit 45 illustrated in FIG. The high-resolution image update part 456 which repeats the update process of the high-resolution image in the super-resolution calculating part 45 in the 1st example of 2 times is provided. The high-resolution image update unit 456 is provided with the low-resolution actual image Fa that is the current frame from the AFE 2 and the low-resolution actual images Fb and Fc that have been subjected to the positional shift correction by the positional shift correction unit 44. A high resolution image Fx1 estimated by the initial high resolution estimation unit 451 is given. As in the first example described above, the pixel values of the low-resolution real images Fa to Fc with u pixels are set to Ya to Yc, and the pixel value of the high-resolution image Fx1 with v pixels is set to X.

  As in the case of the first example, the super-resolution calculation unit 45 in this example is the initial high-resolution estimation unit 453 in which the low-resolution real image Fa from the AFE 2 and the positional deviation correction unit 44 have been subjected to positional deviation correction. The initial high resolution image Fx1 is generated by the low resolution real images Fb and Fc. Then, the pixel value X of the initial high resolution image Fx1 and the pixel values Ya to Yc of the low resolution actual images Fa to Fc are given to the ultra high resolution image updating unit 456.

  First, the super high-resolution image update unit 456 has the gradient ∂I / ∂X (= 2 × {WaT · (Wa · X−Ya) + WbT · (Wb · X−Yb) + WcT · (Wc · X−Yc). ) + ΓCT · C · X}). Next, based on the gradient （I / ∂X, a pixel value X1 (= X−∂I / ∂X) of a new high-resolution image Fx2 is obtained. Thereafter, the gradient ∂I / ∂X1 (= 2 × {WaT · (Wa · X1−Ya) + WbT · (Wb · X1−Yb) + WcT · (Wc · X1−Yc) + γCT · C · X1) with respect to the pixel value X1 }) Is subtracted from the pixel value X1 to obtain the pixel value X2 (= X1-∂I / ∂X1) of the high-resolution image Fx3 by two update operations. .

  The ultra-high resolution image update unit 456 can be configured by a FIR (Finite Impulse Response) filter. A filter coefficient calculation method when the ultra-high resolution image update unit 456 is configured by an FIR filter will be described below with reference to the drawings. In order to simplify the description, it is assumed that the super-resolution calculation unit 45 generates a high-resolution image having a resolution that is twice that of the low-resolution real image. Further, the low-resolution real images Fb and Fc for two frames selected from the frame memory 41 are respectively high-resolution images in either the horizontal direction or the vertical direction with respect to the low-resolution real image Fa serving as the current frame. It is assumed that the amount of motion in pixel units in is generated.

Further, a PSF (Point Spread Function) function (point spread function) for generating a low-resolution estimated image from a high-resolution image is given by a 3 × 3 blur filter as shown in FIG. It is assumed that | CX | ² is “0”. When the pixel of interest in the high-resolution image is xpq (p and q are natural numbers) as shown in FIG. 15B, the coefficient in the PSF blur function by the 3 × 3 matrix shown in FIG. k11 to k33 are respectively multiplied by 3 × 3 pixels x (p−1) (q−1) to x (p + 1) (q + 1) of the high resolution image.

  As shown in FIG. 16A, the low-resolution real image Fb has a horizontal movement amount corresponding to one pixel of the high-resolution image with respect to the low-resolution real image Fa, and FIG. As shown in FIG. 5, the low-resolution real image Fc has a motion amount in the vertical direction by one pixel of the high-resolution image with respect to the low-resolution real image Fa. That is, as shown in FIG. 17A, when the pixel ya11 of the low-resolution real image Fa and the pixel x11 of the initial high-resolution image Fx1 overlap at the center position, as shown in FIG. The pixel yb11 of the low-resolution real image Fb and the pixel x21 of the initial high-resolution image Fx1 overlap at the center position, and as shown in FIG. 17C, the pixel yc11 of the low-resolution real image Fc and the initial height The pixel x12 of the resolution image Fx1 overlaps at the center position.

  Then, as shown in FIG. 17A, the center position of the pixel yapq (p and q are natural numbers) of the low resolution real image Fa is the pixel x (2p-1) (2q-1) of the initial high resolution image Fx1. It overlaps with the center position. Further, as shown in FIG. 17B, the center position of the pixel ybpq (p and q are natural numbers) of the low-resolution real image Fb overlaps the center position of the pixel x2p (2q-1) of the initial high-resolution image Fx1. . Further, as shown in FIG. 17C, the center position of the pixel ycpq (p and q are natural numbers) of the low-resolution real image Fc is the center position of the pixel x (2p-1) 2q of the initial high-resolution image Fx1. Overlap.

(1) Pixel Value of Low-Resolution Estimated Image When the positional relationship between each pixel of the low-resolution actual images Fa to Fc and each pixel of the initial high-resolution image Fx1 is as shown in FIG. 17, the initial high-resolution actual image Fx1 The respective pixel values Apq to Cpq (pixel values based on Wa · X to Wc · X) of the estimated low resolution estimated images Fa1 to Fc1 are expressed by the following equations (1) to (3), respectively.
Apq = k11 · x (2p-2) (2q-2) + k21 · x (2p-1) (2q-2) + k31 · x2p (2q-2)
+ K12 ・ x (2p-2) (2q-1) + k22 ・ x (2p-1) (2q-1) + k32 ・ x2p (2q-1)
+ K13 · x (2p-2) 2q + k23 · x (2p-1) 2q + k33 · x2p2q (1)
Bpq ＝ k11 ・ x (2p-1) (2q-2) + k21 ・ x2p (2q-2) + k31 ・ x (2p + 1) (2q-2)
+ K12 · x (2p-1) (2q-1) + k22 · x2p (2q-1) + k32 · x (2p + 1) (2q-1)
+ K13 · x (2p-1) 2q + k23 · x2p2q + k33 · x (2p + 1) 2q (2)
Cpq = k11 · x (2p-2) (2q-1) + k21 · x (2p-1) (2q-1) + k31 · x2p (2q-1)
+ K12 · x (2p-2) 2q + k22 · x (2p-1) 2q + k32 · x2p2q
+ K13 · x (2p-2) (2q + 1) + k23 · x (2p-1) (2q + 1) + k33 · x2p (2q + 1) (3)

(2) Gradient at each pixel position of the high-resolution image When the pixel values of the low-resolution estimated images Fa1 to Fc1 are obtained as described above, the initial height is determined by the difference from the pixel values of the low-resolution actual images Fa to Fc. The gradient ∂I / ∂X for the resolution image Fx1 is calculated. Hereinafter, calculation methods in three cases in which the target pixel of the high-resolution image is the pixel x (2p-1) (2q-1), x2p (2q-1), and x (2p-1) 2q will be described. .

(2-a) When the pixel of interest is a pixel x (2p-1) (2q-1) As shown in FIG. 18 (a), the pixel x (2p-1) that is the pixel of interest in the initial high-resolution image Fx1. ) Filter processing is performed on the region R by 3 × 3 pixels x (2p−2) (2q−2) to x2p2q centering on (2q−1). Therefore, using the pixel values of the low-resolution real images Fa to Fc and the low-resolution estimated images Fa1 to Fc 1 whose center positions are located in the region R, the gradient ∂I of the pixel x (2p−1) (2q−1) is used. / ∂X is calculated.

That is, as shown in FIG. 18A, in the region R, the pixels x (2p-1) (2q-1), x (2p-2) (2q-1), x2p ( 2q-1), x (2p-1) (2q-2), and x (2p-1) 2q, respectively, a pixel yapq of the low-resolution real image Fa and a pixel yb (p-1) q of the low-resolution real image Fb , Ybpq, and the pixels ycp (q-1) and ycpq of the low-resolution real image Fc are positioned so as to overlap the center position. Therefore, the pixel values yapq, yb (p-1) q, ybpq, ycp (q-1), ycpq in the low-resolution real images Fa-Fc and the pixel values Apq, B (p- in the low-resolution estimated images Fa1-Fc1 1) Using q, Bpq, Cp (q-1), Cpq and the blur function shown in FIG. 15, the gradient ∂I / ∂ of the pixel x (2p-1) (2q-1) X is calculated.
∂I / ∂X (x (2p-1) (2q-1)) ＝ k22 ・ (Apq-yapq) / K1
+ K12 ・ (B (p-1) q-yb (p-1) q) / K1
+ K32 ・ (Bpq-ybpq) / K1
+ K21 ・ (Cp (q-1) -ycp (q-1)) / K1
+ K23 ・ (Cpq-ycpq) / K1 (4)
(K1 = k12 + k22 + k32 + k11 + k13 + k31 + k33)

(2-b) When the target pixel is the pixel x2p (2q-1) As shown in FIG. 18B, in the initial high resolution image Fx1, the pixel x2p (2q-1) as the target pixel is the center. Filter processing is performed on the region R formed by 3 × 3 pixels x (2p−1) (2q−2) to x (2p + 1) 2q. Therefore, the gradient ∂I / ∂X of the pixel x2p (2q-1) is calculated using the pixel values of the low-resolution actual images Fa to Fc and the low-resolution estimated images Fa1 to Fc1 whose center positions are located in the region R. Is done.

That is, as shown in FIG. 18B, within the region R, the pixels x (2p-1) (2q-1), x (2p + 1) (2q-1), x2p ( 2q-1), x (2p-1) (2q-2), x (2p-1) 2q, x (2p + 1) (2q-2), x (2p + 1) 2q Pixels yapq and ya (p + 1) q of the image Fa, pixels ybpp of the low-resolution real image Fb, pixels ycp (q-1), ycpq and yc (p + 1) (q-1) of the low-resolution real image Fc , Yc (p + 1) q are positioned so as to overlap the center position. Therefore, the pixel values yapq, ya (p + 1) q, ybpq, ycp (q-1), ycpq, yc (p + 1) (q-1), yc (p + 1) in the low resolution real images Fa-Fc. ) q and pixel values Apq, A (p + 1) q, Bpq, Cp (q-1), Cpq, C (p + 1) (q-1), C (p) in the low resolution estimated images Fa1 to Fc1 The gradient ∂I / ∂X of the pixel x2p (2q-1) is calculated by (+1) q and the blur function shown in FIG.
∂I / ∂X (x2p (2q-1)) = k22 ・ (Bpq-ybpq) / K2
+ K12 ・ (Apq-yapq) / K2
+ K32 ・ (A (p + 1) q-ya (p + 1) q) / K2
+ K11 ・ (Cp (q-1) -ycp (q-1)) / K2
+ K13 ・ (Cpq-ycpq) / K2
+ K31 ・ (C (p + 1) (q-1) -yc (p + 1) (q-1)) / K2
+ K33 ・ (C (p + 1) q-yc (p + 1) q) / K2 (5)
(K2 = k12 + k22 + k32 + k11 + k13 + k31 + k33)

(2-c) When the pixel of interest is the pixel x (2p-1) 2q As shown in FIG. 18C, the pixel x (2p-1) 2q that is the pixel of interest is centered in the initial high-resolution image Fx1. A filtering process is performed on the region R including the 3 × 3 pixels x (2p−2) (2q−1) to x2p (2q + 1). Therefore, using the pixel values of the low-resolution real images Fa to Fc and the low-resolution estimated images Fa1 to Fc 1 whose center positions are located in the region R, the gradient ∂I / ∂X of the pixel x (2p-1) 2q is Calculated.

That is, as shown in FIG. 18C, in the region R, the pixels x (2p-1) (2q-1), x (2p-1) (2q + 1), x ( 2p-2) (2q-1), x2p (2q-1), x (2p-2) (2q + 1), x2p (2q + 1), x (2p + 1) 2q Fa pixels yapq, yap (q + 1), low resolution real image Fb pixels yb (p-1) q, ybpq, yb (p-1) (q + 1), ybp (q + 1), low resolution Each pixel ycpq of the actual image Fc is positioned so as to overlap the center position. Therefore, the pixel values yapq, yap (q + 1), yb (p-1) q, ybpq, yb (p-1) (q + 1), ybp (q + 1), ybp (q + 1), ycpq and pixel values Apq, Ap (q + 1), B (p-1) q, Bpq, B (p-1) (q + 1), Bp (q + 1) in the low resolution estimated images Fa1 to Fc1 , Cpq and the blur function shown in FIG. 15, the gradient ∂I / ∂X of the pixel x2p (2q-1) is calculated as in equation (6).
∂I / ∂X (x (2p-1) 2q) = k22 ・ (Cpq-ycpq) / K3
+ K21 ・ (Apq-yapq) / K3
+ K23 ・ (Ap (q + 1) -yap (q + 1)) / K3
+ K11 ・ (B (p-1) q-yb (p-1) q) / K3
+ K31 ・ (Bpq-ybpq) / K3
+ K13 ・ (B (p-1) (q + 1) -yb (p-1) (q + 1)) / K3
+ K33 ・ (Bp (q + 1)-ybp (q + 1)) / K2 (6)
(K2 = k21 + k22 + k23 + k11 + k13 + k31 + k33)

(3) Update at each pixel position of the high-resolution image When the gradient ∂I / ∂X at each pixel position of the high-resolution image is calculated as described above, this gradient is used to calculate each pixel value of the initial high-resolution image Fx1. Is subtracted from the pixel value of the updated high-resolution image. That is, the pixel value at the pixel xpq is updated with the gradient ∂I / ∂X (xpq), whereby the pixel value x1pq (= xpq−∂I / ∂X (xpq)) of the high-resolution image Fx2 is calculated. It will be.

Thus, the high-resolution image is updated by performing the processing operations (1) to (3) described above. Therefore, at the time of the second update, the pixel values of the updated high resolution image Fx2 are substituted when the pixel values of the low resolution estimated images Fa2 to Fc2 are obtained by the above equations (1) to (3). That is, the respective pixel values Apq to Cpq (pixel values based on Wa · X1 to Wc · X1) of the low resolution estimated images Fa2 to Fc2 are respectively expressed by the following equations (7) to (9).
Apq = k11 · x1 (2p-2) (2q-2) + k21 · x1 (2p-1) (2q-2) + k31 · x1 (2p) (2q-2)
+ K12 ・ x1 (2p-2) (2q-1) + k22 ・ x1 (2p-1) (2q-1) + k32 ・ x1 (2p) (2q-1)
+ K13 · x1 (2p-2) (2q) + k23 · x1 (2p-1) (2q) + k33 · x1 (2p) (2q)… (7)
Bpq ＝ k11 ・ x1 (2p-1) (2q-2) + k21 ・ x1 (2p) (2q-2) + k31 ・ x1 (2p + 1) (2q-2)
+ K12 ・ x1 (2p-1) (2q-1) + k22 ・ x1 (2p) (2q-1) + k32 ・ x1 (2p + 1) (2q-1)
+ K13 · x1 (2p-1) (2q) + k23 · x1 (2p) (2q) + k33 · x1 (2p + 1) (2q)… (8)
Cpq = k11 · x1 (2p-2) (2q-1) + k21 · x1 (2p-1) (2q-1) + k31 · x (2p) (2q-1)
+ K12 x1 (2p-2) (2q) + k22 x1 (2p-1) (2q) + k32 x1 (2p) (2q)
+ K13 · x1 (2p-2) (2q + 1) + k23 · x (2p-1) (2q + 1) + k33 · x (2p) (2q + 1)… (9)

  When the pixel values of the low-resolution estimated images Fa2 to Fc2 are obtained in this way, the gradient ∂I / ∂X1 for each pixel of the high-resolution image is calculated by the above-described equations (4) to (6). The gradient ∂I / ∂X1 is subtracted from the pixel value of the high resolution image Fx1, whereby the second update process is performed to generate the high resolution image Fx2.

  As described above, when the calculation by the PSF blur function using the 3 × 3 matrix is performed, in the first update processing operation, the pixel of interest xpq in the high resolution image Fx is compared with the pixel of interest in the initial high resolution image Fx1. Is located within a pixel region R (see FIG. 19A) composed of 3 × 3 pixels x (p−1) (q−1) to x (p + 1) (q + 1) centered on The pixel values of the low resolution actual image and the low resolution estimated image are required.

  Further, as described above, the pixel value of the low resolution estimated image can be obtained by substituting 3 × 3 pixels in the initial high resolution image Fx1 into the PSF blur function. In order to acquire a pixel of the low resolution estimated image located at a pixel position other than the target pixel xpq in the pixel region R, a pixel value of 3 × 3 pixels in the initial high resolution image Fx1 is used. Therefore, in the first update processing operation, as shown in FIG. 19B, 5 × 5 pixels x (p−2) (q−2) to x (p + 2) (q) in the initial high resolution image Fx1. +2) will be used.

  That is, in order to update the initial high resolution image Fx1 once, for the target pixel xpq in the initial high resolution image Fx1, 5 × 5 pixels x (p−2) (q−2) in the initial high resolution image Fx1. To x (p + 2) (q + 2) and 3 × 3 pixels x (p−1) (q−1) to x (p + 1) (q + 1) in the initial high-resolution image Fx1 The pixel values of the pixels in the low-resolution real images Fa to Fc located in the pixel region R constituted by

  When the high-resolution image Fx2 obtained by updating in this way is updated, the process is performed using the updated pixel values. That is, the pixel value of 5 × 5 pixels x1 (p−2) (q−2) to x1 (p + 2) (q + 2) in the high resolution image Fx2 with respect to the target pixel x1pq in the high resolution image Fx2. And the low-resolution actual image Fa located in the pixel region R constituted by 3 × 3 pixels x1 (p−1) (q−1) to x1 (p + 1) (q + 1) in the high-resolution image Fx2 The pixel value of the pixel in Fc is required.

  However, when the pixel values are updated from the initial high resolution image Fx1 to the high resolution image Fx2, for each pixel, the pixel value of 5 × 5 pixels in the initial high resolution image Fx1 and the 3 × 3 in the initial high resolution image Fx1 The pixel values of the pixels in the low-resolution real images Fa to Fc located in the pixel region R composed of pixels are required. That is, 5 × 5 pixels x1 (p−2) (q−2) to x1 (p + 2) (q + 2) in the high resolution image Fx2 used for the target pixel x1pq in the high resolution image Fx2 respectively. In the first update process, the pixel value of 5 × 5 pixels in the initial high-resolution image Fx1 and the low-resolution actual pixel value located in the pixel region R composed of 3 × 3 pixels in the initial high-resolution image Fx1 It is calculated using the pixel values of the pixels in the images Fa to Fc.

  Therefore, when the initial high-resolution image Fx1 is updated twice, 5 × 5 pixels x (p−2) (q−2) to x (p + 2) (q + 2) for each pixel of interest xpq Further updating is performed using the updated pixel value in. That is, the pixel value of 5 × 5 pixels in the initial high resolution image Fx1 with the 5 × 5 pixels x (p−2) (q−2) to x (p + 2) (q + 2) as the center, and the initial value The pixel values of the pixels in the low-resolution real images Fa to Fc located in the pixel region R configured by 3 × 3 pixels in the high-resolution image Fx1 are used. In this way, when the initial high resolution image Fx1 is updated twice, as shown in FIG. 20, the 9 × 9 pixels x (in the initial high resolution image Fx1 are compared with the target pixel xpq in the initial high resolution image Fx1. p-4) (q-4) to x (p + 4) (q + 4) and 7 × 7 pixels x (p-3) (q-3) to x ( The pixel values of the pixels in the low-resolution real images Fa to Fc located in the pixel region R1 constituted by p + 3) (q + 3) are required.

  Therefore, the FIR filter constituting the ultra-high resolution image update unit 456 is located in the pixel region R1 including the pixel value of 9 × 9 pixels in the initial high resolution image Fx1 and the 7 × 7 pixels in the initial high resolution image Fx1. The pixel value of the pixel in the low-resolution real images Fa to Fc to be input is input and the pixel value of the pixel of interest is output.

  As for the coefficients of the FIR filter, first, as described above, the pixel value of the low resolution estimated image obtained by the equations (1) to (3) is substituted into the equations (4) to (6), and the gradient is obtained. Ask for. Then, the pixel value of the high resolution image updated by the gradient obtained by the equations (4) to (6) is substituted into the equations (7) to (9), and the pixel value of the low resolution estimated image obtained thereby is calculated. Substituting into equations (4) to (6), the second gradient is obtained. Then, by developing a subtraction formula used when updating with the gradient obtained by the equations (4) to (6), the coefficient multiplied by the pixel value of 9 × 9 pixels in the initial high resolution image Fx1 and the initial high The coefficient multiplied by the pixel value of the pixel in the low-resolution real images Fa to Fc located in the pixel region R1 configured by 7 × 7 pixels in the resolution image Fx1 can be acquired as the coefficient of the FIR filter.

  In this example, the high-resolution update amount calculation unit 453 calculates the update amount when the update operation repeated in the super-resolution operation unit 45 in the first example is performed twice. However, the update amount when the update operation in the super-resolution calculation unit 45 in the first example described above is repeated three or more times may be calculated. It is assumed that the updating operation in the super-resolution calculation unit 45 is repeated H times and the PSF blur function is composed of a (2K + 1) × (2K + 1) matrix. At this time, the FIR filter constituting the high-resolution update amount calculation unit 453 has a pixel value of (4H × K + 1) × (4H × K + 1) pixels in the initial high-resolution image Fx1 and (2 (2H) in the initial high-resolution image Fx1. −1) × K + 1) × (2 (2H−1) × K−1) pixels and pixel values of the pixels in the low-resolution real images Fa to Fc located in the pixel region R1 are input, and the target pixel The pixel value of is output. Further, in this example, like the super-resolution calculation unit 45 in the first example, the super-resolution calculation process may be repeated as a configuration including the selection unit 452 and the frame memory 455.

  In the above description, the image processing method according to the present invention has been described using the image pickup apparatus having the configuration shown in FIG. 1 as an example. The image processing method according to the present invention can also be used in a display device that performs processing. FIG. 21 shows a display device including an image processing apparatus (corresponding to an “image processing unit”) that performs the image processing method according to the present invention.

  The display device shown in FIG. 21 is similar to the image pickup device shown in FIG. 1 in that the image processing unit 4, the expansion processing unit 8, the display unit 9, the audio output circuit unit 10, the speaker unit 11, the timing generator 12, the CPU 13, the memory 14, An operation unit 15 and bus lines 16 and 17 are provided. A tuner unit 21 that selects a broadcast signal received externally, a demodulator unit 22 that demodulates a broadcast signal selected by the tuner unit 21, and an interface to which a compressed signal that is a digital signal input from the outside is input. 23.

  In the case of receiving a broadcast signal, the display device of FIG. 21 selects a broadcast signal of a desired channel by the tuner unit 21, and then demodulates the broadcast signal by the demodulation unit 22, thereby compressing by the MPEG compression coding method. A digital signal is obtained as a signal. When this digital signal is given to the decompression processing unit 8, the digital signal which is a compressed signal is subjected to decompression processing by the MPEG compression coding method.

  Then, when the operation unit 15 instructs to increase the resolution of the image, the image signal obtained by the expansion processing by the expansion processing unit 8 is given to the image processing unit 4 to select the above-described low-resolution real image. By performing the processing and the super-resolution processing, a high resolution image is generated. Thereafter, an image signal based on the generated high-resolution image is given to the display unit 9, and image reproduction is performed. Also, the audio signal obtained by the expansion processing of the expansion processing unit 8 is given to the speaker unit 11 through the audio output circuit unit 10, so that the sound is reproduced and output.

  Further, in the imaging device of FIG. 1 or the display device of FIG. 21, the image that is subjected to the super-resolution processing by the image processing unit 4 may be a moving image or a still image. In the above description, each processing operation has been described focusing on the operation when an image signal to be a moving image is input.

  In the above description, the low-resolution real image selection process for performing the super-resolution processing has been described for the case of a one-dimensional image. An example of real image selection processing will be described below. In the following embodiments, for the sake of simplicity of explanation, operations in super-resolution processing when the resolution is doubled will be described. That is, in each of the following embodiments, a low resolution for performing super-resolution processing in the image processing unit 4 when performing super-resolution processing on an image signal that is a two-dimensional image acquired from the image sensor 1. An example of an actual image selection processing operation will be described.

  The selection processing operation of the low resolution real image in the present embodiment will be described with reference to FIG. FIG. 22 is a diagram illustrating the relationship between the pixel positions after positional deviation correction of a plurality of low-resolution real images including the target frame. In the present embodiment, a case where a low-resolution real image of two frames including a target frame is selected from a low-resolution real image of three frames including the target frame will be described. Then, the low-resolution real image of the target frame is F1, the low-resolution real images of the other two frames are F2 and F3, and the pixel positions in the low-resolution real image F1 are indicated by black dots in FIG. A point by a triangle represents a pixel position in the low-resolution real image F2, and a point by a black square represents a pixel position in the low-resolution real image F3.

  First, as shown in FIG. 22, the reference position for selecting the low-resolution real images F2 and F3 is set to the center position O of four adjacent pixels in the low-resolution real image of the frame of interest F1 (in FIG. (Indicated by a circle). In FIG. 22, the distance between the pixel position and the center position O is the distance d2 in the low-resolution real image F2, and the distance d3 (d3> d2) in the low-resolution real image F3. In FIG. 22, a position where the distance from the center position O is d2 is indicated by a circle centered on the center position O indicated by a dotted line.

  Then, a low-resolution real image whose pixel position is close to the center position O serving as a selection criterion is selected as a low-resolution real image used for super-resolution processing. That is, since the distance d2 from the pixel position center position O in the low resolution real image F2 is shorter than the distance d3 from the pixel position center position O in the low resolution real image F3, the low resolution real image F2 is super-resolved. A low-resolution real image suitable for processing is selected. Therefore, a high-resolution image is acquired by performing super-resolution processing by the low-resolution real image F1 used as the attention frame and the selected low-resolution real image F2.

  The low resolution real image selection processing operation in this embodiment will be described with reference to FIG. FIG. 23 is a diagram illustrating the relationship between the pixel positions after the positional deviation correction of the low-resolution real images of a plurality of frames including the target frame, as in FIG. Also in the present embodiment, similarly to the first embodiment, a case where a two-frame low-resolution real image including the target frame is selected from the three low-resolution real images F1 to F3 including the target frame will be described. In FIG. 23, the pixel positions of the low-resolution real images F1 to F3 are represented in the same format as in FIG.

In the present embodiment, unlike the case of the first embodiment, in each of the low-resolution real images F2 and F3, the pixel position is a position separated by an equal distance d from the center position O serving as a selection reference. Then, the pixel position of the low-resolution real image F2 is located at a position away from the center position O by dx2 in the horizontal direction and dy2 in the vertical direction, and the pixel position of the low-resolution real image F3 is at the center position O. On the other hand, it is assumed that it is located at a position separated by dx3 (| dx2 | <| dx3 |) in the horizontal direction and dy3 (| dy2 |> | dy3 |) in the vertical direction. The distance d is d = ((dx2) ² + (dy2) ² ) ^1/2 = ((dx3) ² + (dy3) ² ) ^1/2 .

  Thus, since the distance between the pixel position of each of the low-resolution real images F2 and F3 and the center position O is equal to d, the low-resolution real images F2 and F3 are selected when selected based on the distance to the center position O. Either of these can be selected. Therefore, in order to select one of the low-resolution real images F2 and F3, in this embodiment, a low-resolution real image that further shortens the horizontal distance from the center position O is selected. That is, in the example of FIG. 23, the horizontal distance | dx2 | between the pixel position of the low-resolution actual image F2 and the center position O is the horizontal distance between the pixel position of the low-resolution actual image F3 and the center position O. Since the distance in the direction is shorter than | dx3 |, the low-resolution real image F2 is selected as the low-resolution real image used for the super-resolution processing.

  As described above, in this embodiment, when there are a plurality of low resolution real images in which the distance between the reference position such as the center position O and the pixel position is equal, the distance in the horizontal direction between the reference position and the pixel position is short. Select a resolution real image. As a result, a super-resolution process can be performed by selecting a low-resolution real image that increases the resolution in the horizontal direction with respect to the visual characteristic of high visibility in the vertical direction.

  The low resolution real image selection processing operation in this embodiment will be described with reference to FIG. FIG. 24 is a diagram illustrating the relationship between the pixel positions after the positional deviation correction of the low-resolution real images of a plurality of frames including the target frame, as in FIG. Also in the present embodiment, similarly to the first embodiment, a case where a two-frame low-resolution real image including the target frame is selected from the three low-resolution real images F1 to F3 including the target frame will be described. In FIG. 24, the pixel positions of the low-resolution real images F1 to F3 are represented in the same format as in FIG.

  In the present embodiment, as in the case of the second embodiment, the pixel positions of the low-resolution real images F2 and F3 are separated by the same distance d from the center position O serving as a selection reference. On the time axis, the low resolution real images F1 to F3 are acquired in the order of F1, F2, and F3, and the distances between the pixel positions of the low resolution real images F1, F2 (low resolution real images F1, F2 The amount of motion between F2) d4 is longer than the distance between the pixel positions of F2 and F3 of the low-resolution actual image (the amount of motion between the low-resolution actual images F2 and F3) d5.

  Thus, in the present embodiment as well, as in the second embodiment, the distance between the pixel position of each of the low-resolution real images F2 and F3 and the center position O is equal to d, and thus is based on the distance from the center position O. In this case, both the low resolution real images F2 and F3 can be selected. Therefore, in order to select one of the low-resolution real images F2 and F3, in this embodiment, the amount of motion between the frames is further compared, and a frame with a small amount of motion is selected. That is, in the example of FIG. 24, since the distance d5 between the pixel positions of F2 and F3 of the low-resolution real image is shorter than the distance d4 between the pixel positions of F1 and F2 of the low-resolution real image, super-resolution The low-resolution real image F3 is selected as the low-resolution real image used for processing.

  As described above, in this embodiment, when there are a plurality of low-resolution real images having the same distance between the reference position such as the center position O and the pixel position, the amount of motion between the low-resolution real images that are temporally continuous is determined. Select a smaller low-resolution real image. That is, by selecting a low-resolution real image that has a small amount of motion with the low-resolution real image that is the previous frame, it is possible to select a low-resolution real image that is less affected by the amount of motion and perform super-resolution processing. it can.

  Note that, in the case where there are a plurality of low-resolution real images in which the distance between the reference position and the pixel position is equal, the above-described second and third embodiments are the conditions for selecting the low-resolution real image used for the super-resolution processing. However, it may be possible to select these conditions.

  In addition, when the resolution is increased, if the enlargement ratio differs between the vertical direction and the horizontal direction, the distance between the reference position and the pixel position becomes shorter with respect to the direction in which the enlargement ratio is high (the number of interpolation pixels is large). A low-resolution real image may be selected as a low-resolution real image suitable for super-resolution processing. That is, when the horizontal enlargement ratio is m times and the vertical enlargement ratio is n (m> n) times, the enlargement ratio is larger in the horizontal direction, and therefore the relationship is as shown in FIG. In the horizontal direction, the low-resolution real image F2 in which the distance between the reference position and the pixel position is shortened is selected.

  The present invention can be applied to an imaging device or a display device that includes an image processing device that increases the resolution of an image by super-resolution processing.

These are the block diagrams which show the internal structure of the imaging device provided with the image process part used as the image processing apparatus of this invention. These are diagrams showing the relationship between the luminance distribution of a subject and image data in frames with different shooting times. These are the figures for demonstrating the outline | summary of the flow in a super-resolution process. These are diagrams showing the relationship between the luminance distribution of a subject and image data in frames with different shooting times. These are the figures for demonstrating the selection process of the low-resolution real image in the super-resolution process which makes a resolution 2 times. These are the figures for demonstrating the selection process of the low-resolution real image in the super-resolution process which makes a resolution 3 times. These are figures which show the example of a relationship with the reference | standard in the selection process of the low-resolution real image in the super-resolution process which makes a resolution 3 times. These are figures which show the relationship of the detection area in a representative point matching method. These are figures which show the relationship between the representative point of a reference | standard image, and the sampling point of a non-reference | standard image. FIG. 4 is a diagram illustrating a relationship between a representative point of a reference image and a sampling point of a non-reference image when detecting a motion amount within one pixel. FIG. 10 is a diagram illustrating a relationship between pixel values of representative points of a reference image and sampling points of a non-reference image when detecting a motion amount within one pixel. These are block diagrams which show the internal structure of the image processing part in the imaging device shown in FIG. FIG. 13 is a block diagram illustrating a first example of an internal configuration of a super-resolution calculation unit in the image processing unit illustrated in FIG. 12. FIG. 13 is a block diagram showing a second example of the internal configuration of the super-resolution calculation unit in the image processing unit shown in FIG. 12. These are figures which show the relationship between the coefficient of a PSF blur function, and the pixel position of a high resolution image. These are diagrams showing the positional relationship between the low-resolution actual images Fb and Fc and the low-resolution actual image Fa that is the current frame. These are figures which show the positional relationship of each of the low-resolution real images Fa-Fc and the initial high-resolution image Fx1. These are figures which show the pixel value used with respect to the attention pixel at the time of calculating a gradient with respect to the initial high-resolution image Fx1. FIG. 4 is a diagram illustrating a pixel range in a high-resolution image used in the first update processing operation. FIG. 5 is a diagram illustrating a pixel range in a high-resolution image used in two update processing operations. These are block diagrams which show the internal structure of the display apparatus provided with the image processing part used as the image processing apparatus of this invention. These are the figures which show the relationship between 3 frames for demonstrating the selection processing operation | movement of the low resolution image in Example 1. FIG. These are the figures which show the relationship between 3 frames for demonstrating the selection processing operation | movement of the low resolution image in Example 2. FIG. These are the figures which show the relationship between 3 frames for demonstrating the selection processing operation | movement of the low resolution image in Example 3. FIG.

Explanation of symbols

1 Image sensor 2 AFE
Reference Signs List 3 microphone 4 image processing unit 5 audio processing unit 6 compression processing unit 7 driver unit 8 expansion processing unit 9 display unit 10 audio output circuit unit 11 speaker unit 12 timing generator 13 CPU
DESCRIPTION OF SYMBOLS 14 Memory 15 Operation part 16, 17 Bus line 20 External memory 41 Frame memory 42 Motion amount calculation part 43 Image evaluation part 44 Position shift correction part 45 Super-resolution operation part 46 Signal processing part

Claims (9)

In an image processing apparatus including a high-resolution part that generates a high-resolution image from a first low-resolution image serving as a frame of interest and a second low-resolution image of M (an integer of 1 or more) frame
A motion amount detection unit that detects a motion amount between each of the third low resolution images of N (an integer satisfying N> M) frames and the first low resolution image;
In consideration of the amount of motion in the third low-resolution image obtained by the motion amount detection unit, a reference set based on the pixel position of the first low-resolution image from the third low-resolution image of N frames An image evaluation unit that selects the third low-resolution image of the M frame whose pixel position is closer to the position as the second low-resolution image;
With
The image processing apparatus, wherein the third low-resolution image that has been made the second low-resolution image by the image evaluation unit is provided to the high-resolution unit, and the high-resolution image is generated.   The image processing apparatus according to claim 1, wherein the reference position is set to a position that is an intermediate position between adjacent pixel positions in the vertical direction and the horizontal direction in the first low-resolution image.   The image processing apparatus according to claim 1, wherein the reference position is set to a pixel position that is an interpolation pixel for the first low-resolution image in the high-resolution image.   In the image evaluation unit, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, an intermediate between adjacent pixel positions in the vertical direction and the horizontal direction in the first low-resolution image. The image processing apparatus according to claim 3, wherein the third low-resolution image whose pixel position is close to a position to be a position is selected as the second low-resolution image.   In the image evaluation unit, when the third low resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the third low resolution with a short distance in the horizontal direction between the pixel position and the reference position. The image processing apparatus according to claim 1, wherein an image is selected as the second low-resolution image.   In the image evaluation unit, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the third low-resolution image having a small amount of motion with respect to the previous frame temporally adjacent is selected. The image processing apparatus according to claim 1, wherein the image processing apparatus is selected as the second low-resolution image.   In the image evaluation unit, when the third low-resolution image having the same distance between the reference position and the pixel position is a plurality of frames, the resolution enlargement ratios in the horizontal direction and the vertical direction are different in the super-resolution step. The low-resolution image having a short distance in the direction in which the enlargement ratio between the pixel position and the reference position is large is selected as the second low-resolution image. The image processing apparatus described. In an electronic device provided with an image signal based on an image that becomes a plurality of frames by external input or imaging, and having a high resolution function that converts an image based on the image signal into a high resolution image,
The image processing unit according to any one of claims 1 to 7 is provided as an image processing unit that realizes the high resolution function, and the high resolution processing is performed using the image based on the image signal as the low resolution image. An electronic apparatus, wherein the desired high-resolution image is generated. In an image processing method comprising a high-resolution step of generating a high-resolution image from a first low-resolution image serving as a frame of interest and a second low-resolution image of M (an integer of 1 or more) frame,
A motion amount detecting step for detecting a motion amount between each of the third low resolution images of N (an integer satisfying N> M) frames and the first low resolution image;
Considering the amount of motion in the third low-resolution image obtained from the motion amount detection step, a reference set from the third low-resolution image of N frames based on the pixel position of the first low-resolution image A low-resolution image selection step of selecting the third low-resolution image of the M frame whose pixel position is closer to the position as the second low-resolution image;
With
In the high-resolution step, the high-resolution image is generated based on the third low-resolution image and the first low-resolution image that are made the second low-resolution image in the low-resolution image selection step. An image processing method.

Images