JP2009194896A - Image processing device and method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the quality of a high-resolution image obtained by super-resolving processing. <P>SOLUTION: A super-resolving portion (50) for creating a high-resolution image from a plurality of low-resolution images by super-resolving processing, includes a blur-amount estimation portion (52) for estimating the magnitude of blur contained in each low-resolution image, a reference image setting portion (53) for setting as a reference image a low-resolution image whose magnitude of blur is the smallest based on the estimation results, and setting the other low-resolution images as comparison images, and a position-shift detection portion (54) for finding position shift amounts between the reference image and the comparison images with sub-pixel accuracy. By using the found position shift amounts and the plurality of low-resolution images, and performing resolution enhancement using the reference image as the reference, a higher-resolution image is created. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing, an image processing method, and an imaging apparatus that uses them. In particular, the present invention relates to a super-resolution processing technique for generating a high-resolution image from a plurality of low-resolution images.

  Super-resolution processing has been proposed as an image processing technique for increasing the resolution of a low-resolution image. A device that performs super-resolution processing refers to a plurality of low-resolution images with positional deviation, and based on the amount of positional deviation between the plurality of low-resolution images and the image data of the plurality of low-resolution images, By performing the conversion, one high-resolution image is generated. This high resolution is performed with reference to a reference image selected from a plurality of low resolution images, and the amount of misalignment is calculated with reference to the reference image. For this reason, the content (including image quality and composition) of the obtained high-resolution image greatly depends on the reference image. The reference image is also called a reference frame, a target frame, a target image, or the like.

  In general, the reference image is an image generated earliest in an image sequence arranged in a time series obtained by continuous photographing (see, for example, Patent Document 1).

JP 2005-197910 A JP 2007-151080 A

  As described above, the content of the high-resolution image obtained by increasing the resolution greatly depends on the reference image. Therefore, in order to obtain a more desirable high-resolution image, it is necessary to devise a method for selecting the reference image.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing method, and an imaging apparatus that contribute to generation of a desirable high-resolution image.

  An image processing apparatus according to the present invention is an image processing apparatus that generates a high-resolution image having a higher resolution than a resolution of the low-resolution image from a plurality of low-resolution images, based on image information of the plurality of low-resolution images. A reference image setting means for selecting a reference image from the plurality of low-resolution images, and executing a resolution increasing process for increasing the resolution with respect to the plurality of low-resolution images based on the reference image; The high-resolution image is generated.

  Specifically, for example, the image processing apparatus further includes a blur amount estimation unit that estimates a blur size included in each low-resolution image based on the image information, and the reference image setting unit includes the blur amount The reference image is selected based on the estimation result of the estimation means.

  This is expected to improve the image quality of the resulting high resolution image.

  More specifically, for example, the blur amount estimation means estimates the size of blur included in each low resolution image based on the amount of high frequency components in each low resolution image.

  Alternatively, specifically, for example, the plurality of low-resolution images form a low-resolution image sequence arranged in time series obtained by sequential shooting by an imaging device, and the blur amount estimation unit includes the low-resolution image sequence in the low-resolution image sequence Detecting a shake state of the imaging device during the shooting period of the low-resolution image sequence based on a positional deviation amount between temporally adjacent low-resolution images, and from the detection result of the shake state, Estimating the magnitude relationship of the blur size between low resolution images.

  For example, the reference image setting unit selects a low resolution image having the smallest estimated blur size as the reference image from among the plurality of low resolution images.

  If an image with a small blur is selected as a reference image that serves as a reference for increasing the resolution, the image quality of the obtained high-resolution image can be improved as compared with the case where the reference image is randomly selected.

  Further, for example, each low-resolution image is obtained by photographing with an imaging device, and the image information includes detection data of a sensor that detects a blurring state of the imaging device during an exposure period of each low-resolution image, and the reference image The setting means selects the reference image based on the detection data of the sensor.

  For example, the reference image setting unit selects, as the reference image, a low-resolution image having the smallest blurring of the imaging device during the exposure period among the plurality of low-resolution images.

  If an image with a small blur of the imaging device during the exposure period is selected as a reference image that becomes a reference for high resolution, the image quality of the obtained high resolution image can be improved compared to the case where the reference image is selected at random. Can be improved.

  In addition, for example, the image processing apparatus further includes subject size detection means for detecting a size on the image of a specific subject included in each low-resolution image based on the image information, and the reference image setting means includes: The reference image may be selected based on the detected size of the specific subject.

  Alternatively, for example, the image processing apparatus further includes a face state detection unit that detects a human face from each low-resolution image based on the image information and detects the face state, and the reference image setting unit includes: The reference image may be selected based on the detection result of the face state detection means.

  More specifically, for example, the reference image setting unit sets a low resolution image other than the reference image among the plurality of low resolution images as a reference image, and the image processing apparatus includes the reference image and the reference image. A positional deviation detecting means for obtaining a positional deviation amount with respect to a reference image with a resolution higher than a pixel interval of the standard image, and using the obtained positional deviation quantities and the plurality of low resolution images, with the standard image as a reference A high-resolution image generation unit configured to generate the high-resolution image by performing the high-resolution processing.

  According to a first imaging apparatus of the present invention, an imaging unit that acquires a plurality of low-resolution images arranged in time series by sequential shooting, and a high-resolution image generated by high-resolution processing on the plurality of low-resolution images. And an image processing apparatus.

  The second image pickup apparatus according to the present invention includes an image pickup unit having an image pickup element and an optical system for forming an optical image corresponding to a subject on the image pickup element to obtain a shot image by shooting, and the shot image A focus evaluation value deriving unit for deriving a focus evaluation value based on a video signal in a focus evaluation region provided in the image pickup, and performing autofocus control by driving and controlling the optical system based on the focus evaluation value In the apparatus, an image for generating a high-resolution image having a resolution higher than the resolution of the low-resolution image from the plurality of low-resolution images by referring to the plurality of captured images obtained from the imaging unit as a plurality of low-resolution images. A processing device, wherein the image processing device selects a reference image from the plurality of low-resolution images based on the focus evaluation value derived for each low-resolution image. Comprising a setting means, based on the reference image, and generates the high resolution image by performing a resolution enhancement process to increase the resolution with respect to the plurality of low-resolution images.

  As a result, it is possible to generate a high-resolution image having contents more desirable for the user.

  Specifically, for example, in the autofocus control, the optical system is driven and controlled so that the focus evaluation value takes a maximum value, and the reference image setting unit is configured to perform a maximum alignment among the plurality of low resolution images. A low resolution image corresponding to the focus evaluation value is selected as the reference image.

  As a result, a low-resolution image having a composition in which the subject of interest of the user is arranged in the focus evaluation area can be easily selected as the reference image. As a result, it is possible to generate a high-resolution image having a composition that is more desirable for the user.

  More specifically, for example, in the second imaging device, the reference image setting unit sets a low-resolution image other than the reference image among the plurality of low-resolution images as a reference image, and the image processing device Uses a displacement detection means for obtaining a displacement amount between the reference image and the reference image with a resolution higher than a pixel interval of the reference image, and uses the obtained displacement amounts and the plurality of low-resolution images. And high-resolution image generation means for generating the high-resolution image by performing the high-resolution processing on the basis of the reference image.

  An image processing method according to the present invention is an image processing method for generating a high resolution image having a resolution higher than the resolution of the low resolution image from a plurality of low resolution images, based on image information of the plurality of low resolution images. A reference image is selected from the plurality of low-resolution images, and the high-resolution image is generated by executing, on the plurality of low-resolution images, high-resolution processing that increases the resolution based on the reference image. It is characterized by doing.

  Note that the image information of the low-resolution image according to the present invention includes image data representing the luminance and / or color of each pixel of the low-resolution image, and may further include the detection data described above.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus, the image processing method, and imaging device which contribute to the production | generation of a desirable high resolution image can be provided.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to eighth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera, for example. The imaging device 1 can shoot moving images and still images, and can also shoot still images simultaneously during moving image shooting. Note that the moving image shooting function may be omitted, and the imaging apparatus 1 may be a digital still camera capable of shooting only a still image.

[Description of basic configuration]
The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and an SDRAM (Synchronous Dynamic Random Access Memory). ), An external memory 18 such as an SD (Secure Digital) card or a magnetic disk, a decompression processing unit 19, a VRAM (Video Random Access Memory) 20, an audio output circuit 21, and a TG (timing generator). ) 22, a CPU (Central Processing Unit) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25.

  The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and provides the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the internal memory 17 during signal processing as necessary.

  The imaging unit 11 includes an optical system, an aperture, and a driver (not shown in FIG. 1) in addition to the imaging element (image sensor) 33. Incident light from the subject enters the image sensor 33 via the optical system and the stop. Each lens constituting the optical system forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

  The image sensor 33 is composed of, for example, a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 33 photoelectrically converts an optical image incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the imaging device 33 includes a plurality of pixels (light receiving pixels; not shown) that are two-dimensionally arranged in a matrix, and in each photographing, each pixel receives a signal charge having a charge amount corresponding to the exposure time. store. The electrical signal from each pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to the subsequent AFE 12 in accordance with the drive pulse from the TG 22.

  The AFE 12 amplifies the analog signal output from the image sensor 33, converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The degree of amplification of signal amplification in the AFE 12 is controlled by the CPU 23. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The video signal is composed of a luminance signal Y representing the luminance of the image and color difference signals U and V representing the color of the image.

  The microphone 14 converts the peripheral sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. The compressed video signal is recorded in the external memory 18 at the time of shooting and recording a moving image or a still image. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting and recording, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being correlated with each other in time by the compression processing unit 16, and those after compression are externally transmitted. Recorded in the memory 18.

  The recording button 26a is a push button switch for instructing the start / end of moving image shooting and recording, and the shutter button 26b is a push button switch for instructing shooting and recording of a still image.

  The operation modes of the imaging apparatus 1 include a shooting mode in which moving images and still images can be shot, and a playback mode in which moving images and still images stored in the external memory 18 are played back and displayed on the display unit 27. Transition between the modes is performed according to the operation on the operation key 26c. In the shooting mode, shooting is sequentially performed at a predetermined frame period, and an image sequence arranged in time series is acquired from the image sensor 33. Each image forming this image sequence is called a “frame image”.

  When the user presses the recording button 26a in the shooting mode, under the control of the CPU 23, the video signal of each frame image and the corresponding audio signal obtained after the pressing are sequentially sent via the compression processing unit 16 to the external memory 18. To be recorded. When the user presses the recording button 26a again after starting the moving image shooting, the recording of the video signal and the audio signal in the external memory 18 is finished, and the shooting of one moving image is completed. In the shooting mode, when the user presses the shutter button 26b, a still image is shot and recorded.

  When the user performs a predetermined operation on the operation key 26c in the reproduction mode, a compressed video signal representing a moving image or a still image recorded in the external memory 18 is expanded by the expansion processing unit 19 and then written to the VRAM 20. It is. In the shooting mode, normally, video signals are generated sequentially by the video signal processing 13 regardless of the operation contents of the recording button 26a and the shutter button 26b, and the video signals are sequentially written in the VRAM 20.

  The display unit 27 is a display device such as a liquid crystal display, and displays an image corresponding to the video signal written in the VRAM 20. In addition, when a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded in the external memory 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

  The video signal processing unit 13 is configured to be able to perform super-resolution processing in cooperation with the CPU 23. Through the super-resolution processing, one high-resolution image is generated from a plurality of low-resolution images. The video signal of the high resolution image can be recorded in the external memory 18 via the compression processing unit 16. The resolution of the high resolution image is higher than that of the low resolution image, and the number of pixels in the horizontal and vertical directions of the high resolution image is larger than that of the low resolution image. For example, when an instruction to capture a still image is given, a plurality of frame images as a plurality of low resolution images are acquired, and a high resolution image is generated by performing super-resolution processing on them. Alternatively, for example, the super-resolution processing is performed on a plurality of frame images as a plurality of low resolution images obtained at the time of moving image shooting.

  The basic concept of super-resolution processing will be briefly described. As an example, super-resolution processing using a reconstruction method will be described. FIG. 2 is a conceptual diagram of super-resolution processing using a MAP (Maximum A Posterior) method, which is a kind of reconfigurable method. In this super-resolution processing, one high-resolution image is estimated from a plurality of low-resolution images obtained by actual shooting, and the original plurality of low-resolution images are estimated by degrading the estimated high-resolution image. To do. A low resolution image obtained by actual photographing is particularly called an “observation low resolution image”, and an estimated low resolution image is particularly called an “estimated low resolution image”. Thereafter, the high resolution image and the low resolution image are repeatedly estimated so that the error between the observed low resolution image and the estimated low resolution image is minimized, and the finally acquired high resolution image is output.

  FIG. 3 is a flowchart showing the flow of super-resolution processing. First, in step S11, an initial high resolution image is generated. This initial high resolution image is generated from a reference image described later. In the subsequent step S12, the original observed low resolution image for constructing the current high resolution image is estimated. The estimated image is referred to as an estimated low resolution image as described above. In the subsequent step S13, an update amount for the current high resolution image is derived based on the difference image between the observed low resolution image and the estimated low resolution image. This update amount is derived so that an error between the observed low-resolution image and the estimated low-resolution image is minimized by repeatedly executing the processes in steps S12 to S14. In the subsequent step S14, the current high resolution image is updated using the updated amount, and a new high resolution image is generated. Thereafter, the process returns to step S12, and the newly generated high resolution image is regarded as the current high resolution image, and the processes of steps S12 to S14 are repeatedly executed. Basically, as the number of repetitions of each process of steps S12 to S14 increases, the resolution of the obtained high-resolution image is substantially improved, and an ideal high-resolution image is obtained.

  Super-resolution processing based on the above-described operation flow is performed in the imaging apparatus 1. The super-resolution processing performed in the imaging apparatus 1 may be any type of super-resolution processing, but in the present embodiment, a description will be given of a case where a reconfiguration type super-resolution processing is adopted. To do. The reconfiguration method includes an ML (Maximum-Likelihood) method, a MAP (Maximum A Posterior) method, a POCS (Projection Onto Convex Set) method, an IBP (Iterative Back Projection) method, and the like. In this embodiment, the MAP method is used. Take the configuration using as an example.

  As examples relating to the super-resolution processing, first to eighth examples will be described below. The matters described in one embodiment can be applied to other embodiments as long as no contradiction arises.

<< First Example >>
First, the first embodiment will be described. FIG. 4 is an internal block diagram of the super-resolution unit 50 responsible for the super-resolution processing. Reference numerals 51, 52, and 54 to 57 refer to the respective parts provided in the video signal processing unit 13 of FIG. 1, and the first, second, third, and fourth frame memories 61, 62, 63, and 64 are provided. Are provided in the internal memory 17 of FIG. The function of the reference image setting unit 53 is realized by the CPU 23. However, the function of the reference image setting unit 53 may be realized by the video signal processing unit 13. Hereinafter, the first, second, third, and fourth frame memories are simply referred to as frame memories. The frame memories 61 to 63 are memories for storing image data of frame images as observed low resolution images, and the frame memory 64 is a memory for storing image data of high resolution images. FIG. 5 is a diagram showing the flow of operation of each part of FIG. 4 in time series, and the horizontal direction of FIG. 5 corresponds to the time direction. Note that the image data of a noticed image is data (for example, an RGB signal or a YUV signal) representing the luminance and color of the noticed image.

In the first embodiment, a case where one high-resolution image is generated from three observed low-resolution images is taken as an example. The three observed low-resolution images are three frame images taken in succession. These three observation low-resolution images are represented by L ₁ , L _2, and L ₃ , and the observation low-resolution images L ₁ , L _2, and L ₃ are taken in this order. In the following description, (the same is true for L ₄ described later or the like) that the observed low-resolution image L _1, is L ₂ and L ₃ each image L _1, L also ₂ and L ₃ and the abbreviated. It is assumed that a positional shift caused by camera shake or the like occurs between two different images among the images L ₁ , L ₂ and L ₃ .

Since shooting is performed in the order of the images L ₁ , L _2, and L ₃ , first, image data representing the image L ₁ is input to the selection unit 51 and the blur amount estimation unit 52, and then image data representing the image L _2. There are input, the input is further image data representing the image L ₃ to the next. The selection unit 51 selects a frame memory in which image data of the input observation low-resolution image is stored. Specifically, the selection unit 51 sends the image data of the image L ₁ , the image data of the image L ₂ , and the image data of the image L _{3 to} the frame memories 61, 62, and 63 and stores them.

The blur amount estimation unit 52 is a part for estimating the size of blur (hereinafter referred to as blur amount) included in the input image, and calculates a blur amount evaluation value corresponding to the blur amount. Blur estimation unit 52 first calculates the blur evaluation value corresponding to the blur of the image L ₁ on the basis of the image data of the image L _1, the image L ₂ and then based on the image data of the image L ₂ A blur amount evaluation value corresponding to the blur amount is calculated, and then a blur amount evaluation value corresponding to the blur amount of the image L ₃ is calculated based on the image data of the image L ₃ . Each blur amount evaluation value is sequentially transmitted to the reference image setting unit 53.

  When the imaging apparatus 1 is supported by hand, so-called camera shake can act on the casing of the imaging apparatus 1 during the exposure period of the observed low-resolution image. In addition, so-called subject shake may occur. Subject blur means blurring of a subject on an image caused by the subject moving in real space during the exposure period. When camera shake or subject blur occurs in the observed low-resolution image, the entire observed low-resolution image or part of the observed low-resolution image is blurred due to camera shake or subject blur. The blur amount estimation unit 52 estimates the size of the blur derived from the camera shake or the subject blur based on the image data of the observed low resolution image.

  The blur amount estimation unit 52 estimates the blur amount using a characteristic that, for example, when a camera shake or a subject blur occurs, a high frequency component in the image is attenuated. That is, a predetermined high frequency component in the observed low resolution image is extracted, and the amount of blur is estimated based on the amount of the extracted high frequency component. The amount of the high frequency component is also referred to as the intensity of the high frequency component.

  With reference to FIG. 6, a method of calculating the blur amount evaluation value for one observation low-resolution image will be described. FIG. 6 is an internal block diagram of a blur amount estimation unit that can be used as the blur amount estimation unit 52. The blur amount estimation unit in FIG. 6 includes an extraction unit 71, an HPF (high pass filter) 72, and an integration unit 73.

  The extraction unit 71 is provided with a video signal of the observed low resolution image. The extraction unit 71 extracts the luminance signal in the evaluation area defined in the observed low resolution image from the video signal. The evaluation area is, for example, the entire area in the observed low resolution image. However, it is also possible to set a partial area in the observed low-resolution image (for example, a main subject area described in a fifth embodiment described later) as an evaluation area. The HPF 72 extracts only a predetermined high frequency component from the luminance signal extracted by the extraction unit 71. For example, the HPF 72 is formed by a Laplacian filter having a 3 × 3 filter size as a spatial filter as shown in FIG. 7, and spatial filtering is performed so that the Laplacian filter acts on each pixel in the evaluation region. Then, output values corresponding to the filter characteristics of the Laplacian filter are sequentially obtained from the HPF 72. The HPF 72 may be formed by a frequency filter, and a high frequency component may be extracted on the frequency domain using Fourier transform.

  The integrating unit 73 integrates the magnitude of the high frequency component extracted by the HPF 72 (that is, the absolute value of the output value of the HPF 72), and outputs the integrated value as a blur amount evaluation value. The blur amount evaluation value for a certain observed low resolution image increases as the blur amount in the observed low resolution image decreases.

Three observation low-resolution images L ₁ ~L ₃ when generating one high-resolution image by the resolution of the reference one of the three images L ₁ ~L ₃ is a high-resolution And the other two images are set as reference images. The reference image can be read as a reference frame or a reference frame image, and the reference image can be read as a reference frame or a reference frame image. The amount of displacement necessary for generating a high resolution image is calculated with reference to the reference image. The reference image is also used as a source for generating an initial high-resolution image. For this reason, the image quality of the finally obtained high-resolution image greatly depends on the reference image. Therefore, the standard image setting unit 53 performs processing for setting the standard image and the reference image based on the blur amount evaluation value. Specifically, among the observed low-resolution images L _{1 to} L ₃ , an observed low-resolution image estimated to have the smallest blur is set as a reference image, and the remaining two observed low-resolution images are used as reference images. Set. When the amount of blur is small, the blur amount evaluation value becomes large. Therefore, the observation low-resolution image having the largest blur amount evaluation value is set as the reference image.

Now, the number of the observed low resolution image corresponding to the standard image is represented by a, and the number of the observed low resolution image corresponding to the two reference images is represented by b and c. When the reference image is the image L ₁ , a = 1 and b = 2 and c = 3, and when the reference image is the image L ₂ , a = 2, b = 1 and c = 3, Is image L ₃ , a = 3, b = 1 and c = 2. After setting the reference image, the reference image setting unit 53 in FIG. 4 performs an observation to be read out by the misregistration detection unit 54, the initial high-resolution image generation unit 55, and the super-resolution processing unit 57 based on the values a, b, and c. Control the order of low-resolution images. That is, they control which frame memory image data should be read at which timing.

  The misregistration detection unit 54 calculates the misregistration amount between two observation low-resolution images using a representative point matching method, a block matching method, a gradient method, or the like. The amount of displacement calculated here has a so-called sub-pixel resolution having a resolution higher than the pixel interval of the observed low-resolution image. That is, the amount of positional deviation is calculated with a distance shorter than the interval between two adjacent pixels in the observed low resolution image as the minimum unit. The positional deviation amount is a two-dimensional amount including a horizontal component and a vertical component, and is also called a motion amount or a motion vector.

The misregistration detection unit 54 calculates the misregistration amount between the standard image and each reference image using the standard image as a standard. Therefore, after setting the reference image, the position shift detecting unit 54, first observed image data of the low resolution image L _a and L _b are input, relative to the observed low-resolution image L _a, the observed low-resolution image L _a A displacement amount V _ab between L and L _b is calculated (see FIG. 5). Next, the positional shift detection unit 54, the observed image data of the low resolution image L _a and L _c are input, relative to the observed low-resolution image L _a, between the observed low-resolution image L _a and L _c position deviation amount V _ac of is calculated.

On the other hand, after setting the reference image, the initial high resolution image generation unit 55 (hereinafter, abbreviated as generator 55) generates an initial high resolution image H ₀ on the basis of a reference image observed low-resolution image L _a (See FIG. 5). This generation process corresponds to the process in step S11 of FIG. The initial high resolution image H ₀ corresponds to an initial image of a high resolution image to be finally generated. Hereinafter, the initial high resolution image H ₀ may be simply referred to as a high resolution image H ₀ or an image H ₀ . For example, to generate an image with an increased number of pixels in the horizontal and vertical directions of the observed low-resolution image L _a using linear interpolation or bicubic interpolation as the image H _0. The generated image data of the image H ₀ is stored in the frame memory 64.

The selection unit 56 in FIG. 4 selects one of the image generated by the generation unit 55 (that is, the image H ₀ ) and the image stored in the frame memory 64, and exceeds the image data of the selected image. This is given to the resolution processing unit 57. Immediately after the image H ₀ is generated, the image H ₀ is selected by the selection unit 56, and after a high resolution image different from the image H ₀ is generated by the super-resolution processing unit 57, it is stored in the frame memory 64. The selected image is selected by the selection unit 56.

In addition, the super-resolution processing unit 57, observed from the frame memory 61 to 63 low-resolution image L _a, the image data of the L _b and L _c are input. Based on the MAP method, the super-resolution processing unit 57 uses the image H ₀ , the observed low-resolution images L _a , L _b and L _c and the misregistration amounts V _ab and V _ac and generates an estimated low-resolution image. An update amount for the image H ₀ is obtained. This process corresponds to the first process of steps S12 and S13 (see FIG. 3). Then, the image H ₀ is updated with the update amount to generate a high-resolution image H ₁ (hereinafter sometimes abbreviated as image H ₁ ). This corresponds to the first step S14 (see FIG. 3). Note that after the image H ₁ is generated, only the lines indicated by bold lines in FIG. 4 function significantly.

The generated image data of the image H ₁ is overwritten and stored in the frame memory 64 and is input again to the super-resolution processing unit 57 via the selection unit 56. The super-resolution processing unit 57 may update the image H ₁ by a method similar to that used to generate the image H ₁ from the image H ₀ to abbreviate the high-resolution image H ₂ (hereinafter abbreviated as image H ₂ ). ) Is generated. The image data of the image H ₂ is overwritten and stored in the frame memory 64. The process of generating the image H ₂ from the image H ₁ corresponds to the second process of steps S12 to S14.

As described above, the super-resolution processing unit 57 repeatedly executes the arithmetic processing including the processes of steps S12 to S14 for updating the high-resolution image and generating a new high-resolution image. Hereinafter, this calculation processing is referred to as super-resolution calculation processing. When n is a natural number, a high resolution image H _n is generated from the high resolution image H _{n −1} by the n-th super-resolution calculation process. In the following description, n is an integer of 0 or more.

FIG. 8 shows an internal block diagram of the super-resolution processing unit 57. Now, the matrix-represented high-resolution image H _n is represented by x ⁿ , and the matrix-represented observed low-resolution image L _k is represented by y _k . That is, for example, the matrix x ⁿ is a line-up of pixel values of pixels forming the high resolution image H _n . k takes a value of 1, 2 or 3.

Then, the matrix x ^{n + 1} with respect to the high-resolution image H _{n + 1} is derived according to the following formula (A-1), high-resolution image H _{n + 1} are produced by this derivation. The second term on the right side of Expression (A-1) represents the update amount for the high-resolution image H _n that should be calculated by the super-resolution processing unit 57. k _NUM represents the number of observed low-resolution images, and in the present example, k _NUM = 3. By updating the high resolution image based on the equation (A-1), the evaluation function I in the MAP method represented by the following equation (A-2) is minimized. X in the formula (A-2) represents a high-resolution image expressed in a matrix when the super-resolution calculation process is performed a certain number of times.

The product of the matrix W _k and the matrix x ⁿ represents an estimated low resolution image as an estimated image of the observed low resolution image L _k . W _k is a matrix for generating an estimated image of the observed low-resolution image L _k from the high-resolution image H _{n, the} amount of positional deviation calculated by the positional deviation detection unit 54, and from the high-resolution image to the low-resolution image. It is an image conversion matrix including a point spread function representing image blur caused by the reduction in resolution and downsampling from a high resolution image to a low resolution image. (Y _k −W _k x ⁿ ) represents a difference image between the observed low resolution image L _k and the estimated low resolution image corresponding to the estimated image. Note that the matrix with the subscript T represents the transposed matrix of the original matrix. Thus, for example, W _k ^T represents a transposed matrix of the matrix W _k .

  C is a matrix for regularization, and α is a regularization parameter. The matrix C is set based on prior knowledge that “a high-resolution image has few high-frequency components”, for example, and is formed by a Laplacian filter expressed in a matrix. Β is a parameter for controlling the feedback amount.

  When the number of iterations of the super-resolution calculation process in the super-resolution processing unit 57 reaches the specified number, the latest high-resolution image obtained by the specified number of super-resolution calculation processes is the high resolution that should be finally obtained. The image is output from the super-resolution processing unit 57 as an image. If it is determined that the update amount for the latest high-resolution image is sufficiently small and the update amount has converged regardless of the number of times of super-resolution calculation processing, the latest high-resolution image is finally obtained. The super-resolution processing unit 57 may output the high-resolution image to be obtained.

  If an image with a large amount of blur is set as a reference image, the accuracy of the calculated positional deviation amount is deteriorated. In the super-resolution processing, the resolution is increased based on the calculated amount of misregistration, so that the accuracy degradation of the misregistration amount leads to the image quality degradation of the high resolution image. If an image with a large amount of blur is set as a reference image, the image quality of the initial high-resolution image to be obtained (ie, the initial high-resolution image) is deteriorated, making it difficult to obtain an ideal high-resolution image. . In this embodiment, an image with a small amount of blur is selected as the reference image, so that the image quality of the obtained high-resolution image is improved.

  Using the image 201 in FIG. 9A having a relatively large amount of blur and the image 202 in FIG. 9B having a relatively small amount of blur, the effect of selecting an image having a small amount of blur as a reference image was verified by experiments. . In this experiment, three images are generated by reducing the resolution of the prepared ideal high-resolution image by a thinning process with a reduction ratio of 50%, and one of the three images is intentionally blurred. That was the reason why the image was greatly blurred. Then, two images with less blur are treated as two observed low-resolution images and one image with more blur is treated as one observed low-resolution image, and the resolution is twice as high as the observed low-resolution image. The image was reconstructed by super-resolution processing. Two observation low-resolution images with less blur correspond to the image 202 in FIG. 9B, and one observation low-resolution image with more blur corresponds to the image 201 in FIG. 9A. Also, a positional shift was caused between two different observed low-resolution images.

  FIGS. 10A and 10B show high-frequency extracted images 211 and 212 of the images 201 and 202 obtained when the high-pass filter (the Laplacian filter in FIG. 7) is applied to the images 201 and 202, respectively. . In FIGS. 10A and 10B, the pixels with smaller pixel values in the high-frequency extracted image are shown in black. Due to the large blur included in the image 201, the entire high-frequency extracted image 211 is black. When the images 201 and 202 are given to the blur amount estimation unit in FIG. 6, the blur amount evaluation value (HPF integrated value) calculated for the image 201 is about 1/20 of that calculated for the image 202. Met.

  In the experiment, the PSNR (Peak Signal to Noise Ratio) between the high-resolution image output from the super-resolution processing unit 57 in FIG. 4 and the ideal high-resolution image was obtained. As is well known, PSNR is an index representing the similarity between contrasted images, and the higher the similarity, the higher the PSNR. FIG. 11 is a graph showing the dependence of this PSNR on the number of iterations of super-resolution calculation processing. A broken line 221 in FIG. 11 represents the dependence of the PSNR on the number of iterations when an observed low-resolution image with many blurs corresponding to the image 201 is set as a reference image, and a broken line 222 in FIG. This shows the dependence of the PSNR on the number of iterations when a small number of observed low-resolution images are set as reference images. The PSNR is higher by about 2 dB when the image with less blur is set as the reference image than when the image with more blur is set as the reference image. Also from this experiment, the superiority of selecting an image with a small amount of blur as a reference image is understood.

  A supplementary explanation will be given regarding the read / write timing of image data with respect to the internal memory 17 including the frame memories 61 to 64. Reference is made to FIG. 12 (see also FIG. 5). FIG. 12 is a flowchart showing the flow of super-resolution processing with particular attention to image data read / write timing.

First, in step S21, the image data of the observed low resolution images L _{1 to} L ₃ are sequentially written in the internal memory 17. After the reference image is selected from the images L _{1 to} L _{3, the process} proceeds from step S21 to step S22. In step S22, the position shift detection unit 54, reads the image data of the image L _b as the image L _a and the reference image as a reference image from the internal memory 17 to calculate the positional shift amount V _ab, followed by the reference It reads each image data of the image L _c as an image L _a and the reference image as an image from the internal memory 17 and calculates the position deviation amount V _ac with. On the other hand, in step S23, generating unit 55 reads the image data of the image L _a from the internal memory 17 to generate an initial high resolution image. The generated image data of the initial high resolution image is written in the internal memory 17. Then, in subsequent step S24, the image data of the image L ₁ ~L ₃ and the current high-resolution image is read out from the internal memory 17, the high-resolution image is updated by the super-resolution processing described above, and update The image data of the high resolution image is written in the internal memory 17. When the super-resolution calculation process is repeatedly executed, the process of step S24 is repeatedly executed. Note that the same image data may be read from the internal memory 17 in steps S22, S23, and S24 at the same time. For example, it may be performed that simultaneously the image L _a reading of the image data and step S23 in the step S22.

As described above, when the position shift amount calculation process is executed after the reference image is selected from the images L _{1 to} L ₃ , the image data of the observed low resolution image is obtained from the internal memory 17 four times in the step S 22. (4 sheets) must be read (and reading of the image data of the image L _a and L _b, the image data of the image L _a and L _c read).

In order to contribute to the reduction of the number of readings, the reference image selection process and the positional deviation amount calculation process may be performed in parallel. In this case, every time an observed low-resolution image is obtained, a positional deviation amount (V ₁₂ and V ₂₃ described later) is calculated between two temporally adjacent low-resolution images, and a reference image is set. After that, the calculated misregistration amount may be converted into misregistration amounts (V _ab and V _ac described above) between the base image and each reference image.

Specifically, the processing is as follows. After the image L ₁ is acquired and the image data of the image L ₁ is written in the internal memory 17, when the image L ₂ is acquired, the image data of the image L ₂ is written in the internal memory 17 and the position shift detection unit 54. At the same time, the image data of the image L ₁ is read from the internal memory 17 and sent to the misregistration detection unit 54 (at this stage, the first image data reading is executed). Then, the displacement detection unit 54 calculates a displacement V ₁₂ between the image L ₁ and the image L ₂ with the image L ₁ as a reference.

Thereafter, when the image L ₃ is acquired, the image data of the image L ₃ is written in the internal memory 17 and sent to the misalignment detection unit 54. At the same time, the image data of the image L ₂ is read from the internal memory 17 and sent to the misregistration detection unit 54 (at this stage, the second image data reading is executed). Then, the displacement detection unit 54 calculates a displacement V ₂₃ between the image L ₂ and the image L ₃ with the image L ₂ as a reference.

On the other hand, in parallel with the calculation of the positional deviation amounts V ₁₂ and V ₂₃ , the blur amount estimation unit 52 calculates the blur amount evaluation values for the images L ₁ to L _3, and based on the calculated blur amount evaluation values. A reference image is set. The misregistration detection unit 54 determines the misregistration amount V _ab of the reference image when the misregistration amounts V ₁₂ and V ₂₃ are viewed from the standard image, based on which of the images L _{1 to} L ₃ is the set standard image. Convert to V _ac . The operation after V _ab and V _ac are obtained by this conversion is as described above. For example, when V ₁₂ = (0.5, 0.5) and V ₂₃ = (− 0.25, −0.25) and a = 3, b = 1 and c = 2, V _ab = V ₃₁ = V ₃₂ + V ₂₁ = −V ₂₃ −V ₁₂ = (0.25,0.25) − (0.5,0.5) = (− 0.25, −0.25) according to V _ab is determined, V _ac is calculated according to _{V ac = V 32 = -V 23} = (0.25,0.25).

Thus, by performing the selection process of the reference image and the calculation process of the positional deviation amount in parallel, the positional deviation amount V _ab and V _ac are detected because the positional deviation amount conversion process is required. Although the image quality is slightly deteriorated, the number of times of reading out the image data is reduced by 2 times, and the power consumption can be reduced. Note that performing the reference image selection process and the positional deviation amount calculation process in parallel can also be applied to other embodiments described later.

<< Second Example >>
Next, a second embodiment will be described. Although the super-resolution unit 50 of FIG. 4 can also be used in the second embodiment, the blur amount estimation method differs between the first and second embodiments. This difference will be described. Items described in the first embodiment are applied to items not particularly described in the second embodiment.

For convenience of explanation, in the second embodiment, a case where one high-resolution image is generated from four observed low-resolution images is taken as an example. The four observed low-resolution images are four frame images taken in succession. These four observed low-resolution images are represented by L ₁ , L ₂ , L ₃ and L ₄ , and the images L ₁ , L ₂ , L ₃ and L ₄ are taken in this order. Note that the number of frame memories in the super-resolution unit 50 is changed from that in FIG. 4 in view of the fact that the number of observed low-resolution images used for the super-resolution processing is four.

In the second embodiment, the misalignment detector 54 in FIG. 4 sequentially calculates the misalignment amount (so-called interframe motion vector) between two temporally adjacent low-resolution images. In other words, it calculates a position deviation amount V ₁₂ and V ₂₃ described in the first embodiment, the image L ₃ as a reference, the position shift amount V ₃₄ between the image L ₃ and the image L _4. Based on the calculated displacement amounts V ₁₂ , V _23, and V ₃₄ , the blur amount estimation unit 52 is the state of camera shake that has acted on the imaging device 1 during the imaging period of the image sequences L _{1 to} L ₄ (in other words, The shake state of the image pickup apparatus 1 is estimated, and the magnitude relationship between the blur amounts of the observed low-resolution images L _{1 to} L ₄ is estimated based on the estimation result. Based on the estimation result of the blur amount estimation unit 52, the reference image setting unit 53 sets the observed low resolution image corresponding to the minimum blur amount as a reference image, and sets the remaining three observed low resolution images as reference images. To do.

A more specific description will be given with reference to FIG. A curve 240 in FIG. 13 shows a change in the amount of misalignment in the first specific example. In the graphs of FIG. 13 and FIG. 14 to be described later, the horizontal axis represents time and the vertical axis represents the amount of positional deviation, and the horizontal axis also corresponds to the frame number. The frame numbers of the images L _{1 to} L ₄ are 1 to 4, respectively.

Points 241, 242 and 243 are points representing the magnitudes of the positional deviation amounts V ₁₂ , V ₂₃ and V ₃₄ in the first specific example, respectively. The magnitudes of the positional deviation amounts V ₁₂ , V _23, and V ₃₄ that are two-dimensional quantities are expressed by | V ₁₂ |, | V ₂₃ |, and | V ₃₄ |, respectively. The curve 240 is formed using interpolation so that the points 241 to 243 are on the curve 240. A line formed by connecting the magnitudes of the positional deviation amounts V ₁₂ , V _23, and V ₃₄ using an interpolation method, such as a curve 240, is called a camera shake amount locus. The camera shake amount trajectory is the size of the camera shake that has acted on the imaging device 1 during the imaging periods (including the exposure periods) of the images L _{1 to} L ₄ (in other words, the size of the camera shake caused by the camera shake). It can be interpreted that it represents a change in the time). When the inequality “| V ₁₂ |> | V ₂₃ | <| V ₃₄ |” is satisfied as in the example shown in FIG. 13, a camera shake amount locus is drawn by spline interpolation or the like.

It should be noted that, when the hand movement amount trajectory is formed using the interpolation method, in addition to the magnitudes of the positional deviation amounts V ₁₂ , V _23, and V ₃₄ , the frame image L ₀ and the image L that are captured immediately before the image L ₁ is captured. ₁ position shift amount V ₀₁ between, and may be also used a positional deviation amount V ₄₅ between the frame image L ₅ which is immediately after shooting an image L ₄ and the image L _4.

  A point 245 represents a point where the magnitude of the positional deviation amount on the hand movement amount locus 240 is minimized. The blur amount estimation unit 52 specifies the frame number closest to the position of the local minimum point 245 in the time direction as the minimum camera shake frame number. Then, the size of the camera shake that acted during the exposure period of the observed low resolution image corresponding to the minimum camera shake frame number is smaller than that of any other observed low resolution image, and as a result, the observation low resolution corresponding to the minimum camera shake frame number is reduced. It is estimated that the amount of blur of the resolution image (the amount of blur due to camera shake) is smaller than that of any other observed low-resolution image.

The reference image setting unit 53 sets the observed low resolution image corresponding to the minimum camera shake frame number as a reference image, and sets the remaining three observed low resolution images as reference images. In the example of FIG. 13, the point 245 to a position between the point 242 and the point 243, the minimum camera shake frame number observed low-resolution image corresponding to the frame number blur minimum hand with a "3" is L _3. Thus, the image L ₃ is set to the reference image, the image L _1, L ₂ and L ₄ is set to the reference image.

A method of determining the minimum camera shake frame number when the inequality “| V ₁₂ |> | V ₂₃ | <| V ₃₄ |” is not satisfied will be described. A hand movement amount locus 250 in FIG. 14 shows a change in the amount of positional deviation in the second specific example. Points 251, 252, and 253 are points representing the magnitudes of the positional deviation amounts V ₁₂ , V _23, and V ₃₄ in the second specific example, respectively, and are shaken by connecting the points 251 to 253 using an interpolation method. A quantity trajectory 250 is formed.

In the second specific example corresponding to FIG. 14, since the inequality “| V ₁₂ |> | V ₂₃ |> | V ₃₄ |” is established, the shooting periods of the images L _{1 to} L ₄ (the exposure periods thereof) It can be seen that the size of the camera shake that acted on the imaging device 1 gradually decreased. In this case, the minimum camera shake frame number is determined to be “4”. Conversely, when the inequality “| V ₁₂ | <| V ₂₃ | <| V ₃₄ |” is satisfied, the minimum camera shake frame number is determined to be “1”. When the inequalities “| V ₁₂ | <| V ₂₃ |> | V ₃₄ |” and “| V ₁₂ |> | V ₃₄ |” are satisfied, the minimum camera shake frame number is determined as “4”, and the inequalities “ When | V ₁₂ | <| V ₂₃ |> | V ₃₄ | ”and“ | V ₁₂ | <| V ₃₄ | ”are satisfied, the minimum camera shake frame number may be determined as“ 1 ”. The processing content after the minimum camera shake frame number is determined is as described above.

Note that the minimum camera shake frame number may be determined as follows. The sum of the magnitudes of the displacement amounts V ₀₁ and V ₁₂ , the sum of the magnitudes of the displacement amounts V ₁₂ and V ₂₃ , the sum of the magnitudes of the displacement amounts V ₂₃ and V ₃₄ , and the displacement amount V ₃₄ and The sum of the magnitudes of V ₄₅ is obtained in association with the frame numbers 1, 2, 3 and 4, respectively, and the frame number corresponding to the minimum value of these four sums is determined as the minimum camera shake frame number.

<< Third Example >>
The blur amount estimation method described in the first and second embodiments is an example, and the blur amount may be estimated by a different method. A third embodiment will be described as an embodiment showing a modification of the blur amount estimation method. The third embodiment is implemented in combination with the first embodiment.

  For example, the blur amount may be estimated based on the variance in the histogram of the luminance value of the observed low resolution image. When this method is used, the blur estimation unit 52 in FIG. 4 extracts the luminance signal of each pixel of the observed low resolution image and generates a histogram of the luminance value (that is, the value of the luminance signal) of the observed low resolution image. . Then, the variance of the histogram is calculated as a blur amount evaluation value. The generation of the histogram and the calculation of the variance are performed for each observed low resolution image. Note that the luminance signal forming the histogram may be extracted from the entire area of the observed low-resolution image, or may be extracted from a partial area of the observed low-resolution image.

Since the three observation low-resolution images L _{1 to} L ₃ are three frame images taken in succession, their composition is basically the same. For images with the same composition, the greater the camera shake that occurs during the exposure period, the smoother the luminance between adjacent pixels, and the proportion of pixels in the intermediate gradation increases, and the distribution of luminance values in the histogram becomes intermediate gradation. Centralize. The greater the degree of smoothing, the smaller the variance in the histogram and the smaller the blur amount evaluation value. Therefore, it can be estimated that the larger the blur amount evaluation value, the smaller the blur amount of the observed low-resolution image. Therefore, the reference image setting unit 53 in FIG. 4 uses the observed low resolution image corresponding to the largest blur amount evaluation value as the reference among the blur amount evaluation values of the observed low resolution images L _{1 to} L ₃ based on the variance of the histogram. Set as images, and set the remaining two observed low-resolution images as reference images.

  As an example of the observed low resolution image, an image 261 is shown in FIG. 15A and an image 262 is shown in FIG. The image 261 is a clear image, but the image 262 includes a large blur due to a large camera shake during the exposure period of the image 262. FIGS. 16A and 16B show histograms generated for the images 261 and 262, respectively. Contrasting with the histogram of the image 261 (see FIG. 16A), the histogram of the image 262 (see FIG. 16B) shows a concentration of distribution to intermediate gradations. This concentration reduces the variance.

  In addition, any method for estimating the blur amount of an image can be used. For example, the blur amount of the observed low-resolution image may be estimated using the method described in JP-A-11-27574. In this case, a transformed image on a two-dimensional frequency domain is generated by Fourier transforming the observed low resolution image, and the transformed image is projected onto a circle centered on the origin of the frequency coordinates. Then, the blur amount of the observed low-resolution image is estimated from the projection data. Note that “the size of camera shake” in JP-A-11-27574 corresponds to the amount of blur.

<< 4th Example >>
Next, a fourth embodiment will be described. FIG. 17 is an internal block diagram of the super-resolution unit 50a according to the fourth embodiment. Each part referred by the codes | symbols 51, 54-57, and 61-64 provided in the super-resolution part 50a is the same as those of FIG. The function of the reference image setting unit 53a provided in the super-resolution unit 50a is realized by the CPU 23 or the video signal processing unit 13 in FIG. The super-resolution unit 50a and the super-resolution unit 50 according to the first embodiment (FIG. 4) are the same except that the reference image setting method is different. Therefore, a reference image setting method according to the fourth embodiment will be described below. In the fourth embodiment, as in the first embodiment, it is assumed that one high-resolution image is generated from the _three observed low-resolution images L _{1 to} L ₃ .

In the fourth embodiment, the image pickup apparatus 1 is provided with the sensor unit 75 of FIG. The reference image setting unit 53 a sets a reference image from among the observed low resolution images L _{1 to} L ₃ based on the sensor detection data output from the sensor unit 75. The sensor unit 75 detects a shake of the image pickup apparatus 1 (a shake of the casing of the image pickup apparatus 1) due to a camera shake or the like. Specifically, the sensor unit 75 detects the angular velocity in the yaw direction (horizontal direction) of the imaging device 1 and outputs a signal representing the detection result, and the pitch direction (vertical direction) of the imaging device 1. ) And outputs a signal representing the detection result, and output signals of the angular velocity sensors 75A and 75B are output as sensor detection data.

The magnitude and direction of the shake of the image pickup apparatus 1 (the shake of the casing of the image pickup apparatus 1) are indicated by the sensor detection data. The reference image setting unit 53a sets a reference image based on sensor detection data during each exposure period of the observed low resolution images L _{1 to} L ₃ . Specifically, from the sensor detection data during the exposure period of each of the images L _{1 to} L ₃ , the blur magnitude Q ₁ of the imaging device 1 during the exposure period of the image L _{1 and} the exposure period of the image L ₂ are detected. obtains the magnitude Q ₃ of the shake of the image pickup apparatus 1 in the size of the exposure period of the Q ₂ and the image L ₃ of the shake of the image pickup apparatus 1, corresponds to the minimum size of the magnitude Q ₁ to Q ₃ observation The low resolution image is set as a standard image, and the remaining two observed low resolution images are set as reference images. For example, as shown in FIG. 19, when the inequality “Q ₁ > Q ₃ > Q ₂ ” is established, the image L ₂ is set as a reference image, and the images L ₁ and L ₃ are set as reference images.

The size Q ₁ is, for example, a point image with light from the point light source is stationary in the real space, the blur of the image pickup apparatus 1 during the exposure period of the image L _1, a locus drawn on the image L ₁ length ( Or the distance between the start point and the end point of the trajectory). The same applies to the magnitudes Q ₂ and Q ₃ .

Note that it is possible to assume that the super-resolution unit 50a includes a blur amount estimation unit (not shown). In this case, the blur amount estimating section in the super-resolution unit 50a is seeking image L ₁ from the sensor detection data during the exposure period of each of ~L ₃ of the shake of the image pickup apparatus 1 size Q ₁ to Q _3, imaging It is estimated that the blur amount of the corresponding observed low-resolution image is larger as the blur of the device 1 is larger, and the estimation result is given to the reference image setting unit 53a. The reference image setting unit 53a can set a reference image based on the estimation result. Therefore, for example, when the inequality “Q ₁ > Q ₃ > Q ₂ ” is satisfied, the blur amount of the image L _{1 and} the blur amount of the image L ₂ are the largest among the blur amounts of the images L _{1 to} L _3. is estimated to be small, the image L ₁ and L ₃ are set in the reference image with the image L ₂ is set in the reference image by the estimation result is supplied to the reference image setting section 53a.

  Moreover, although the example which comprises the sensor part 75 with an angular velocity sensor was mentioned above, you may comprise the sensor part 75 with the sensor which detects the physical quantity other than the angular velocity which represents the blurring of the imaging device 1. FIG. For example, the sensor unit 75 may be formed by an acceleration sensor that detects the acceleration of the imaging device 1 or an angular acceleration sensor that detects the angular acceleration of the imaging device 1.

Incidentally, when the image L ₁ is captured, the sensor detection data during the exposure period of the image L ₁ together with the image data representing the luminance and color of the image L ₁ is obtained, these data are optionally associated with each other It is recorded in the external memory 18 (see FIG. 1). When those data relating to the image L ₁ are recorded in the external memory 18, for example, one image file including a main body area and a header area is provided in the external memory 18, and the image data is stored in the main body area of the image file. In addition, sensor detection data is stored in the header area. Sensor detection data of the image L ₁ is attached to the image data of the image L _1, imaging conditions of the image L ₁ can be considered to be information representing. The same applies to images other than the image L ₁ (such as L ₂ ).

In this specification, as shown in FIG. 20, information including image data and sensor detection data is referred to as image information. Image information of the image L ₁ has also including an imaging condition of the image L _1, (same for the image L _2, etc.) representing the characteristics of the image L _1.

<< 5th Example >>
Next, a fifth embodiment will be described. The larger the size of the main subject focused on by the photographer on the image, the more detailed the main subject can be expressed on the image. Therefore, it is desirable to provide an image in which the main subject appears larger to users including the photographer. On the other hand, since the high resolution image obtained by the super-resolution processing is generated based on the observed low resolution image as the reference image, if the size of the main subject on the reference image is relatively large, It is also relatively large. Considering this, in the fifth embodiment, an observed low resolution image in which the main subject appears larger is set as the reference image.

FIG. 21 is an internal block diagram of the super-resolution unit 50b according to the fifth embodiment. Each part referred by the codes | symbols 51, 54-57, and 61-64 provided in the super-resolution part 50b is the same as those of FIG. The functions of the reference image setting unit 53b and the subject size detection unit 76 provided in the super-resolution unit 50b are realized by the CPU 23 or the video signal processing unit 13 in FIG. The super-resolution unit 50b and the super-resolution unit 50 according to the first embodiment (FIG. 4) are the same except that the reference image setting method is different. Therefore, hereinafter, functions of the reference image setting unit 53b and the subject size detection unit 76 related to the setting of the reference image will be described. In the fifth embodiment, as in the first embodiment, it is assumed that one high-resolution image is generated from the _three observed low-resolution images L _{1 to} L ₃ .

The subject size detection unit 76 detects the size of the main subject on the observed low resolution image based on the image data of the observed low resolution image for each observed low resolution image. The reference image setting unit 53b sets the observation low resolution image in which the size of the main subject is the largest among the observation low resolution images L _{1 to} L ₃ as the reference image and refers to the other two observation low resolution images. Set to image.

The detection method of the subject size detection unit 76 will be described by giving a specific example. First, as a first detection method, a case where the subject size detection unit 76 is formed so as to be able to perform face detection processing will be described. The face detection process is a process of extracting, from the entire image area of the observed low resolution image, an image area in which image data of a human face exists based on the image data of the observed low resolution image. In the first detection method, subject size detection unit 76 extracts a face region from each of the image L ₁ ~L ₃ by executing the face detection processing for each of the image L ₁ ~L _3. The size of the face area increases as the size of the face present in the face area increases.

In the first detection method, the subject size detection unit 76 sends a face detection result including information representing the size of the face area to the reference image setting unit 53b, and the reference image setting unit 53b is based on the face detection result. An observed low-resolution image from which the largest face area is extracted is set as a reference image. For example, the images 301 to 303 in FIG. 22 are acquired as the images L _{1 to} L ₃ , and the face areas 311 to 313 are extracted from the images 301 to 303, and the size of the face area 312 on the image is the face areas 311 and 313. for larger than the size, the image L ₂ corresponding to the image 302 and the face region 312 is set as the reference image. In the above example, the person's face or the person itself is considered to be the main subject.

  By the way, as one of the causes of acquiring an image sequence having uneven face area sizes such as the images 301 to 303, the imaging apparatus 1 moves along the shooting direction during the shooting period of the images 301 to 303 (that is, Moving back and forth). In such a case, linear transformation (so-called electronic zoom) is performed on the entire images 301 and 303 so that the size of the face area is the same between the images 301 to 303, and the images 301 and 303 after the linear transformation are obtained. Super-resolution processing may be performed using the reference image.

  However, it is also conceivable that a person corresponding to the face areas 311 to 313 moves along the shooting direction instead of the imaging device 1 during the shooting period of the images 301 to 303. Even in such a case, super-resolution processing may be performed using the images 301 and 303 after the linear conversion as reference images. However, when a person moves, the image 302 as the reference image and the linearly-converted image are used. Between the images 301 and 303, the size of the face region matches, but the size of the background region does not match. When super-resolution processing is performed using an image sequence in which the sizes of the background areas are not uniform, a double image can be generated in the background area.

  On the other hand, if such a mismatch occurs, the reliability of the positional deviation amount calculation by the positional deviation detection unit 54 decreases. Therefore, if the reliability is calculated together with the positional deviation amount, the occurrence of the mismatch can be detected. It is. For example, as is well known, the reliability is obtained from data generated when calculating the positional deviation amount (for example, an SSD (Sum of squared difference) value derived when obtaining the positional deviation amount by the block matching method). Is possible.

  Therefore, when the above-mentioned inconsistency is detected, it is determined that the person has moved, and is an image area in the images 301 and 303 after the linear conversion and other than the face area (or the image area where the person's image data exists). The image area may not be used for the super-resolution processing. For example, a high-resolution image is generated using only the entire image area of the image 302 and the face areas 311 and 313 of the images 301 and 303 after linear transformation. By doing so, the face area in the high-resolution image is generated from the face areas of the three low-resolution images, while the portion other than the face area in the high-resolution image is generated only from the image 302. However, it is sufficiently useful because the resolution of the face as the main subject can be increased. Note that the amount of misalignment between the face area 311 of the image 302 and the face areas 311 and 313 of the linearly transformed images 301 and 303, which is necessary for generating a high-resolution image, is obtained when the face detection process is executed. It can be determined from the position of each face region. Such a method of not using a partial image region in the observed low-resolution image after linear conversion for the super-resolution processing is also applicable to a second detection method described later.

  A second detection method by the subject size detection unit 76 will be described. In the second detection method, the subject size detection unit 76 sets a main subject region for each observed low resolution image. The main subject region is an image region that is a partial image region of the observed low-resolution image and in which image data of the main subject exists. For example, considering that there is a high possibility that the main subject is generally present near the center of the image, a preset image region located near the center of the observed low-resolution image is used as the main subject region. Can do. Alternatively, an AF evaluation area described in a seventh embodiment described later may be used as the main subject area. Further alternatively, an image area where image data of a focused subject exists may be used as the main subject area. The position of the image area where the image data of the subject in focus is present can be determined from the high frequency component of the observed low resolution image.

Consider a case where the images 321 to 323 in FIG. 23 are acquired as the images L _{1 to} L _{3 and the} main subject areas 331 to 333 are set for the images 321 to 323. The subject size detection unit 76 sets reference regions 341 to 343 near the center of the main subject regions 331 to 333, respectively. The position and size of the main subject area and the position and size of the reference area are the same between the images 321 to 323.

  Based on the image data of the images 321 to 323, the subject size detection unit 76 determines the color in the reference region 341 of the image 321, the color in the reference region 342 of the image 322, and the color in the reference region 343 of the image 323, respectively. The first to third reference colors are obtained. The color in the reference area 341 is, for example, the average color of each pixel belonging to the reference area 341 (the same applies to the color in the reference area 342 and the color in the reference area 343). When the image data is represented by RGB format signals, the first reference color may be obtained from the R, G, and B signals that are the color signals of the pixels belonging to the reference area 341, and the image data is in the YUV format. In this case, the first reference color may be obtained from the U and V signals that are the color difference signals of the pixels belonging to the reference region 341 (the same applies to the second and third reference colors). . For a pixel, the R, G, and B signals represent the red, green, and blue intensities of that pixel. The first to third reference colors may all be set to the same color. In this case, the first to third reference colors can be set to colors in the reference region 341, 342, or 343.

  The subject size detection unit 76 counts the number of pixels that are the same or similar to the reference color in the main subject region for each observed low resolution image, thereby determining the size of the main subject on the observed low resolution image. Is detected.

Specifically, the following processing is performed for each observed low resolution image. While the position of the reference color in the RGB color space is set as the reference position, the position of the color of each pixel belonging to the main subject area in the RGB color space is detected, and the former position (reference position) and the latter Euclidean distance from the position of. Then, the number of pixels whose Euclidean distance is equal to or smaller than a predetermined threshold value _DTH among the pixels in the main subject area is detected as the size of the main subject. In the example shown in FIG. 23, the number counted for the image L ₂ is larger than those for the images L ₁ and L _3, and as a result, the image L ₂ is set as the reference image.

  According to the fifth embodiment, it is possible to generate a high-resolution image preferable for the user in which the main subject appears large.

<< Sixth Example >>
Next, a sixth embodiment will be described. If an image with a closed eye or an unsatisfactory expression, which is not preferable for the user, is selected as the reference image, the high-resolution image is not preferable. Therefore, in the sixth embodiment, the reference image is set in consideration of the face state.

FIG. 24 is an internal block diagram of the super-resolution unit 50c according to the sixth embodiment. Each part referred by the codes | symbols 51, 54-57, and 61-64 provided in the super-resolution part 50c is the same as those of FIG. The functions of the reference image setting unit 53c and the face state detection unit 77 provided in the super-resolution unit 50c are realized by the CPU 23 or the video signal processing unit 13 in FIG. The super-resolution unit 50c and the super-resolution unit 50 according to the first embodiment (FIG. 4) are the same except that the reference image setting method is different. Therefore, the functions of the reference image setting unit 53c and the face state detection unit 77 related to the setting of the reference image will be described below. In the sixth embodiment, as in the first embodiment, it is assumed that one high-resolution image is generated from the _three observed low-resolution images L _{1 to} L ₃ .

  The face state detection unit 77 performs the face detection process described in the fifth embodiment on each observed low resolution image, and extracts a human face region on the observed low resolution image. Thereafter, the face state detection unit 77 extracts an eye region where an eye exists from the face region based on the image data of the observed low-resolution image, and further performs blink detection processing for detecting the open / closed state of the eye. Any method including a known method can be used for the blink detection processing. For example, by performing template matching using an image representing a standard pupil as a template, the presence or absence of a pupil in the eye region can be detected, and whether or not the eye is open can be detected from the detection result.

Reference image setting unit 53c, the observation of the low-resolution image L ₁ ~L _3, sets the observed low-resolution image is judged with the eye is opened by the blink detection process in the reference image. For example, when it is determined that the eyes on the images L ₂ and L ₃ are open and the eyes on the image L ₁ are closed, the image L ₂ or L ₃ is set as the reference image and the remaining two images ( The images L ₁ and L ₂ or the images L ₁ and L ₃ ) are set as reference images.

When it is determined that the eyes on the images L ₂ and L ₃ are open and the eyes on the image L ₁ are closed, the image L is prevented from appearing on the high resolution image. _It is preferable not to use _one eye area (or the entire face area) for super-resolution processing. In this case, a portion other than the eye region (or the entire face region) of the high resolution image is generated from the images L _{1 to} L ₃ , while the eye region (or the entire face region) of the high resolution image is the images L ₂ and L 3. It will be generated only from ₃ .

  After extracting the face area on the observed low-resolution image, the face state detection unit 77 may perform a smile detection process instead of the blink detection process. The smile detection process is executed for each observed low resolution image. In the smile detection process, it is determined whether or not the face in the face area is a smile based on the image data of the observed low resolution image. Any method including a known method can be used for the smile detection process.

If smile detection process is performed, the reference image setting unit 53c of the observed low-resolution image L ₁ ~L _3, the observed low-resolution image by smile detection process is the face in the face region is determined to be smiling Set to the reference image. For example, when it is determined that the faces on the images L ₂ and L ₃ are smiling and the face on the image L ₁ is not smiling, the image L ₂ or L ₃ is set as a reference image and the remaining two images ( The images L ₁ and L ₂ or the images L ₁ and L ₃ ) are set as reference images.

When it is determined that the faces on the images L ₂ and L ₃ are smiling and the face on the image L ₁ is not smiling, the image L is prevented from appearing on the high-resolution image. _It is better not to use the entire face area of ₁ for super-resolution processing. In this case, the portions other than the face area of the high resolution image are generated from the images L _{1 to} L ₃ , while the face area of the high resolution image is generated only from the images L ₂ and L ₃ .

  It is also possible to set a reference image using both the blink detection process and the smile detection process described above. In this case, an observed low-resolution image in which it is determined that the eyes are open by the blink detection process and the face in the face area is determined to be a smile by the smile detection process may be set as the reference image.

  According to the sixth embodiment, it is possible to generate a high-resolution image that is good for the user and that is favorable for the user.

<< Seventh Embodiment >>
Next, a seventh embodiment will be described. In the seventh embodiment, a reference image is set based on an AF evaluation value used for autofocus control.

  First, with reference to FIG. 25, the internal configuration of the imaging unit 11 of FIG. 1 will be described in detail. FIG. 25 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging element 33, and a driver 34. The optical system 35 includes a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction.

  The driver 34 controls the movement of the zoom lens 30 and the focus lens 31 based on a control signal from the CPU 23, and controls the zoom magnification and the focal position of the optical system 35. Further, the driver 34 controls the opening degree (size of the opening) of the diaphragm 32 based on a control signal from the CPU 23. Incident light from the subject enters the image sensor 33 through the lenses and the diaphragm 32 constituting the optical system 35. Each lens constituting the optical system 35 forms an optical image of the subject on the image sensor 33. A driving pulse for driving the image sensor 33 is given from the TG 22.

  In the imaging apparatus 1, autofocus control is performed by a TTL (Through The Lens) method. In order to perform this autofocus control, the video signal processing unit 13 of FIG. 1 is provided with an AF evaluation unit (focus evaluation value deriving means). FIG. 26 is an internal block diagram of the AF evaluation unit 80 provided in the video signal processing unit 13.

  The AF evaluation unit 80 includes an extraction unit 81, an HPF (high pass filter) 82, and an integration unit 83. The AF evaluation unit 80 calculates one AF evaluation value (focus evaluation value) for one frame image.

  A video signal of a frame image is given to the extraction unit 81. The extraction unit 81 extracts a luminance signal in an AF evaluation area (focus evaluation area) defined in the frame image from the video signal. The AF evaluation area is formed from a plurality of element areas separated from each other. FIG. 27A shows an example of setting the AF evaluation area. For example, nine divided regions are defined in the frame image by dividing the frame image into three equal parts in the vertical direction and the horizontal direction. Then, as shown in FIG. 27A, among the nine divided areas, the whole or a part of the divided area located at the center of the frame image is defined as an element area AR1, and the divided areas located above, below, left and right of the element area AR1. Are all or part of element regions AR2 to AR5. In this case, the total area of the element areas AR1 to AR5 corresponds to the AF evaluation area. It is also possible to form the AF evaluation area with one element area. For example, the element area AR1 itself may be used as the AF evaluation area. In the following description, it is assumed that the AF evaluation area is formed from the element areas AR1 to AR5.

  The HPF 82 extracts only a predetermined high frequency component from the luminance signal extracted by the extraction unit 81. For example, the HPF 82 is formed by a Laplacian filter having a filter size of 3 × 3 as shown in FIG. 7, and spatial filtering is performed so that the Laplacian filter acts on each pixel in the AF evaluation area. Then, output values corresponding to the filter characteristics of the Laplacian filter are sequentially obtained from the HPF 82.

  The accumulating unit 83 accumulates the magnitude of the high frequency component extracted by the HPF 82 (that is, the absolute value of the output value of the HPF 82). This integration is performed individually for each element region. Therefore, each integrated value for the element regions AR1 to AR5 is calculated. The integrating unit 83 calculates an AF evaluation value by weighting and adding each integrated value according to a predetermined weighting coefficient. Assume that the weighting factor for element region AR1 is higher than that for other element regions. For example, as shown in FIG. 27B, consider a case where the weighting coefficient for the element area AR1 is 2.0 and the weighting coefficient for the other element areas AR2 to AR5 is 1.0. Then, the sum of the value obtained by multiplying the integrated value calculated for the element region AR1 by 2 and the integrated value calculated for the element regions AR2 to AR5 is calculated as the AF evaluation value.

  The AF evaluation values calculated for the sequentially obtained frame images are sequentially transmitted to the CPU 23 in FIG. The AF evaluation value for a certain frame image is approximately proportional to the amount of the high frequency component in the AF evaluation region of the frame image, and increases as the amount increases. The amount of the high frequency component is also referred to as the intensity of the high frequency component.

When executing the autofocus control, the AF evaluation values are sequentially obtained while moving the focus lens 31 little by little, and the position of the focus lens 31 at which the AF evaluation value is maximized (or maximum) is obtained as the focus lens position. Then, for example, three frame images taken continuously with the focus lens 31 placed at the focusing lens position are handled as the above-described observed low-resolution images L _{1 to} L ₃ .

The observed low resolution images L _{1 to} L ₃ are also given to the AF evaluation unit 80, and the AF evaluation values for the observed low resolution images L _{1 to} L ₃ are also obtained as described above. In the seventh embodiment, the observed low resolution image corresponding to the maximum AF evaluation value among the AF evaluation values of the observed low resolution images L _{1 to} L ₃ is set as the reference image.

  FIG. 28 is an internal block diagram of a super-resolution unit 50d that performs super-resolution processing according to the seventh embodiment. The super-resolution unit 50d is configured to include each part referred to by reference numerals 51, 80, 53d, and 54 to 57, and frame memories 61 to 64. That is, the super-resolution unit 50d is formed by replacing the blur amount estimation unit 52 and the reference image setting unit 53 in the super-resolution unit 50 of FIG. 4 with the AF evaluation unit 80 and the reference image setting unit 53d. Since the operations of the parts other than the AF evaluation part 80 and the reference image setting part 53d in the super-resolution part 50d are the same as those of the super-resolution part 50 in FIG. 4, the overlapping description for the common part is omitted. The reference image setting unit 53d is realized by the CPU 23. However, the video signal processing unit 13 may have the function of the reference image setting unit 53d.

As the AF evaluation value 80, the AF evaluation values for the observed low resolution images L _{1 to} L ₃ are sent to the reference image setting unit 53d. Reference image setting unit 53d performs processing for setting the reference image and the reference image on the basis of the AF evaluation value for the image L ₁ ~L _3. Specifically, the maximum AF evaluation value among the three AF evaluation values obtained for the images L _{1 to} L ₃ is specified, and the observed low-resolution image corresponding to the maximum AF evaluation value is used as the reference image. On the other hand, the remaining two observation low-resolution images are set as reference images. The operation after the reference image and the reference image are set is the same as that in the first embodiment.

Consider a case where a moving subject is included in the field of view of the imaging apparatus 1. A moving subject means a subject that moves in real space. Then, it is assumed that the frame images 361, 362, and 363 of FIG. 29 are acquired and the frame images 361, 362, and 363 are the observed low resolution images L ₁ , L _2, and L ₃ , respectively. The frame images 361 to 363 include an animal as a moving subject. When the animal moves relative to the imaging device 1, the animal is drawn from the right side in the frame image 361, and the frame image In the image 362, the image is drawn at substantially the center, and in the frame image 363, the image is drawn from the left side. It is assumed that the lens position of the focus lens 31 is fixed so that the animal is in focus when the frame images 361 to 363 are captured.

In this case, as understood from the AF evaluation value calculation method, the AF evaluation value for the image L ₂ is likely to be the maximum among the AF evaluation values for the images L ₁ , L ₂ and L ₃ . When the actual AF evaluation value for the image L ₂ is the largest, since the image L ₂ is set as the reference image super-resolution processing is performed, similar to the image L _2, in the high-resolution image obtained The focused animal will be located approximately in the center.

  According to the present embodiment, a high-resolution image in which a moving subject focused by the photographer (that is, a moving subject in focus) is arranged at the center is obtained. That is, it is possible to automatically obtain a high-resolution image having a desirable composition that matches the photographer's intention.

  In the super-resolution processing in the present embodiment, an initial high resolution image is generated from the reference image using linear interpolation or the like. Then, an estimated low-resolution image is generated from the high-resolution image (initially the initial high-resolution image) according to the amount of positional deviation between the standard image based on the standard image and each reference image, and the estimated low-resolution image and the observed low-resolution image The high resolution image is updated so that the error from the image is minimized. Therefore, it can be said that the finally obtained high-resolution image is an image in which the resolution of the standard image is improved (the reference image is an image that is referred to for improving the resolution). Therefore, when the frame image 362 shown in FIG. 29 is set as the reference image, the focused animal is positioned substantially at the center in the obtained high-resolution image.

<< Eighth Example >>
Next, an eighth embodiment will be described. In the eighth embodiment, elemental techniques for obtaining a high resolution image from a plurality of observed low resolution images will be described in detail. The matters described in the eighth embodiment can be applied to the first to seventh embodiments described above.

  In the eighth embodiment, for simplification of description, a case where one high-resolution image having an image size of 6 × 6 is generated from two observed low-resolution images having an image size of 3 × 3 is taken as an example. . One of the two observed low resolution images is called a first observed low resolution image, and the other is called a second observed low resolution image. It is assumed that the first observed low resolution image is set as a reference image and the second observed low resolution image is set as a reference image. In the following description, the first and second observed low-resolution images may be abbreviated as the first and second observed images, respectively.

30A and 30B show the pixel arrangement of the first and second observation images, and FIG. 30C shows the pixel arrangement of the high-resolution image. The pixel array in FIG. 30C is a pixel array of the high resolution image H _n including the initial high resolution image H ₀ (n is an integer of 0 or more). In addition, an estimated low-resolution image and a difference image are generated from the high-resolution image in the process of the super-resolution calculation process (see FIG. 8), and the pixel array of the estimated low-resolution image and the difference image corresponding to the kth observation image Is the same as that of the k-th observation image (in the eighth embodiment, k is 1 or 2). The estimated low resolution image and difference image corresponding to the kth observed image are hereinafter referred to as the kth estimated low resolution image and difference image, respectively.

  30 (a) to 30 (c) and FIGS. 32, 33 (a) and (b) and FIG. 35 to be described later, the black squares represent the first observed image, the first estimated low-resolution image, or the first image. 1 represents a point at which pixels of the difference image are arranged, and a black-filled triangle represents a point at which pixels of the second observation image, the second estimated low-resolution image, or the second difference image are arranged, and is black A filled circle represents a point where a pixel of a high resolution moving image is arranged.

  In each image, the pixels are arranged in a matrix. Coordinates representing the positions of a total of nine pixels in one observation image (or estimated low-resolution image or difference image) are used using i and j within the range of 0 ≦ i ≦ 2 and 0 ≦ j ≦ 2 ( i, j). However, i and j are integers. Coordinates representing the positions of a total of 36 pixels in the high-resolution image are represented by (i, j) using i and j within the range of 0 ≦ i ≦ 5 and 0 ≦ j ≦ 5. i represents the coordinate value in the row direction (in other words, the vertical direction), and j represents the coordinate value in the column direction (in other words, the horizontal direction).

Further, the first and second observation images expressed in matrix are represented by y ₁ and y ₂ , respectively. Further, as described in the first embodiment, the high-resolution image H _n expressed in a matrix is represented by x ⁿ . The pixel value of the pixel at the coordinates (i, j) in the first observation image is represented by y ₁ (i, j), and the pixel of the pixel at the coordinates (i, j) in the second observation image The value is represented by y ₂ (i, j), and the pixel value of the pixel at the coordinate (i, j) in the high resolution image H _n is represented by x ⁿ (i, j). Then, y ₁ , y ₂ and x ⁿ are represented by the formulas (B-1), (B-2) and (B-3). As described above, the matrix with the subscript T represents the transposed matrix of the original matrix.

[Calculation method of sub-pixel accuracy misalignment]
First, a method for calculating the amount of positional deviation between the first and second observation images using the first observation image as the reference image as a reference will be described. As described in the first embodiment, the positional deviation amount is calculated so as to have sub-pixel resolution by using a representative point matching method, a block matching method, a gradient method, or the like. As an example, a case where the representative point matching method is used will be described.

When the representative point set in the first observation image is a pixel at coordinates (0, 0) and the detection block size in the representative point matching method is 1 × 1, the correlation value corresponding to the position (i, j) E (i, j) is represented by Formula (B-4). Correlation values E (i, j) are obtained for (i, j) = (0,0), (0,1),... (2,2), and a total of nine correlations are obtained. Among the values E (0,0) to E (2,2), the minimum correlation value is specified. If this minimum correlation value is represented by E (i _O , j _O ), the amount of positional deviation between the first and second observation images in integer precision is determined by i _O and j _O.

Decimal portion of the positional deviation amount on the basis of the minimum correlation value E (i _O, j _O) correlation values and correlation values E for vertical and horizontal pixels of the pixel which gives a (i _O, j _O), shown in FIG. 31 for example It calculates using such parabolic approximation. For example, when i _O = 1 and j _O = 1, the row direction component i _SUB and the column direction component j _SUB of the decimal point portion of the misregistration amount are obtained according to the following formulas (B-5) and (B-6). It is done. However, the positional deviation amount of the decimal point portion may be derived using any known method such as linear approximation.

Finally, the amount of positional deviation between the first and second observation images is expressed by i _O and j _O and i _SUB and j _SUB . That is, the row-direction component and the column-direction component (in other words, the vertical component and the horizontal component) of the positional deviation amount are represented by (i _O + i _SUB ) and (j _O + j _SUB ), respectively.

[Method for generating initial high-resolution image]
The initial high-resolution image H ₀ is generated from the first observation image as the reference image using linear interpolation or bicubic interpolation. For example, y ₁ (0,0) = x ⁰ (0,0) is set, and an initial high-resolution image H ₀ is generated by enlarging the first observed image at a magnification of 2 using linear interpolation. In this case, the pixel value x ⁰ (1,3) at the coordinates (1,3) in the image H ₀ represented by the star mark in FIG. 32 is calculated according to the following formula (B-7).

[Method of generating estimated low-resolution image and difference image]
In the super-resolution calculation process executed by the super-resolution processing unit 57 in FIG. 4 and the like, the low-resolution image (the above-described estimated low-resolution image) is reconstructed by applying the matrix W _k to the high-resolution image ( (See FIG. 8). As described above, the matrix W _k is the amount of misregistration calculated by the misregistration detection unit 54, a point spread function (Point Spread Function) representing image blur caused by the reduction in resolution from a high resolution image to a low resolution image, and An image conversion matrix including downsampling from a high resolution image to a low resolution image. As the point spread function included in the matrix W _k , for example, a Gaussian function PSF _k (i, j) represented by the following formula (B-8) is used.

Here, v _k ^x and v _k ^y represent the decimal part of the positional deviation amount of the k-th observation image with respect to the high-resolution image, and v _k ^x and v _k ^y are the row direction components of the positional deviation amount, respectively. And the decimal part of the column direction component. However, v _k ^x and v _k ^y are expressed as values on the high-resolution image. That is, for example, the position shift of the row direction component for 0.5 pixels on the high resolution image is represented by v _k ^x = 0.5.

s _x and s _y represent the size of the spatial filter (the size of the filter stage) that acts on the high resolution image when generating the estimated low resolution image from the high resolution image. s _x and s _y take integer values. In the present example, s _x = s _y = 2. When the value of v _k ^x or v _k ^y does not include a fractional part, the number of taps in the row direction and column direction of the spatial filter is (2s _x +1) and (2s _y +1), respectively, and v _k ^x or When the value of v _k ^y includes a fractional part, the number of taps in the row direction and column direction of the spatial filter is 3s _x and 3s _y , respectively. Σ ² represents the variance in the Gaussian function.

As an example, the integer part of the positional deviation amount of the first and second observation images with respect to the high-resolution image is zero and v ₁ ^x = 0, v ₁ ^y = 0, v ₂ ^x = 0.5, v ₂ Consider that the pixel at the coordinates (1, 1) of the first and second estimated low-resolution images is generated from the high-resolution image when ^y = 0.5.

In this case, since v ₁ ^x = 0 and v ₁ ^y = 0 for the first estimated low resolution image, the number of taps in the row direction and the column direction of the spatial filter applied to the high resolution image is 5 (= 2 × 2 + 1). A square 411 in FIG. 33 (a) is a filter stage (filter impulse response of the filter) to be applied to the high-resolution image to generate the pixel 412 having the coordinates (1, 1) in the first estimated low-resolution image. Is a non-zero range). In order to generate the pixel 412 by reconstruction, the pixel values of the 25 pixels (that is, the pixels at coordinates (0,0) to (4,4)) of the high resolution image located in the filter base 411 are referred to. It will be. The lower part of FIG. 33A shows a point spread function (point spread function based on a low-resolution image) in the matrix W ₁ that acts on the 25 pixels.

For the second estimated low-resolution image, since v ₂ ^x = 0.5 and v ₂ ^y = 0.5, the number of taps in the row direction and the column direction of the spatial filter applied to the high-resolution image is both 6 (= 3 × 2). A square 421 in FIG. 33B indicates a filter stage of a spatial filter that should be applied to the high-resolution image in order to generate the pixel 422 at the coordinate (1, 1) in the second estimated low-resolution image. In order to generate the pixel 422 by reconstruction, the pixel values of 36 pixels (that is, pixels of coordinates (0,0) to (5,5)) of the high resolution image located in the filter base 421 are referred to. It will be. The lower part of FIG. 33B shows a point spread function (point spread function based on a low-resolution image) in the matrix W ₂ that acts on the 36 pixels.

W _k ^xn is a matrix representation of the k-th estimated low-resolution image reconstructed from the high-resolution image H _n , and is represented by the following equation (B-9). FIG. 34 shows an example of the matrix W ₁ . In the present example, the matrix W ₁ is a 9 × 36 matrix. Then, a kth difference image is generated by subtracting each pixel value of the kth estimated low resolution image from each pixel value of the kth observation image (see FIG. 8). A matrix (y _k −W _k x ⁿ ) representing the kth difference image is represented by the following equation (B-10).

[Updating high-resolution images based on difference images]
After the difference image for each low-resolution image is obtained, the high-resolution image is updated by feeding back each difference image onto the coordinate plane of the high-resolution image.

For example, as shown in FIG. 35, the pixels in the first difference image fed back to the pixel 432 at the coordinates (2, 3) on the high resolution image are 6 pixels (black square pixels) in the square frame 431. Yes, the pixels in the second difference image fed back to the pixel 432 are 9 pixels (black triangular pixels) in the square frame 431. However, as described above, the integer part of the positional deviation amount of the first and second observation images with respect to the high-resolution image is zero and v ₁ ^x = 0, v ₁ ^y = 0, v ₂ ^x = 0. 5. Let v ₂ ^y = 0.5.

The feedback value error (2, 3) corresponding to the pixel 432 is expressed by the following formula (B-11). Here, values such as PSF ₁ (0, 1) follow the above formula (B-8). 36 (a) and 36 (b) show point spread functions (point spread functions based on high-resolution images) in the transposed matrices W ₁ ^T and W ₂ ^T used when calculating error (2, 3). Show.

A feedback value is obtained for each pixel forming the high resolution image. The matrix error representing all the obtained feedback values is the sum of products of the matrix (y _k −W _k x ⁿ ) representing the kth difference image and the transposed matrix of W _k as shown in the following formula (B-12). Calculated by calculation. FIG. 37 shows a transposed matrix of the matrix W _{1 in} FIG. After the matrix error is obtained, the matrix x _n is updated according to the equation (B-13) to generate the matrix x _{n + 1} . That is, the high resolution image H _n is updated to generate the high resolution image H _{n + 1} . Formula (B-13) is equivalent to Formula (A-1) described in the first embodiment.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 6 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
Although an example of generating one high-resolution image from two, three, or four low-resolution images has been described above, any number of low-resolution images for generating a high-resolution image can be used as long as the number is two or more. good.

[Note 2]
In the above-described embodiment, the super-resolution processing using the MAP method, which is a kind of reconstruction method, is exemplified as the super-resolution processing. However, what kind of super-resolution processing can be used in the present invention? Super-resolution processing may be used. In the above-described embodiment, after the initial high-resolution image is generated, the super-resolution calculation process including the calculation of the update amount and the update of the high-resolution image based on the update amount is repeatedly performed. The repetition of is not essential, and the high-resolution image H ₁ obtained by performing the super-resolution calculation process only once can be handled as a high-resolution image to be finally obtained.

[Note 3]
In each of the above-described embodiments, various indexes used for selecting the reference image have been individually described. For example, as the index, the blur amount of the low resolution image is used in the first to third embodiments, the sensor detection data is used in the fourth embodiment, and the size of the main subject on the image is used in the fifth embodiment. In the sixth embodiment, the state of the face in the face area (opening / closing state of eyes and facial expression) is used.

  It is also possible to select a reference image using a combination of any two or more of these various indexes. For example, the reference image may be selected based on the amount of blur and the state of the face in the face area. More specifically, for example, when there are a plurality of observed low-resolution images determined to have eyes open by blink detection processing, the amount of blur among the plurality of observed low-resolution images determined to have eyes open The smallest observed low resolution image may be set as the reference image. The method of selecting the reference image based on the amount of blur and the state of the face in the face region is naturally also a method of selecting the reference image based on the amount of blur, and based on the state of the face in the face region. This is also a method of selecting a reference image.

[Note 4]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, a part or all of the arithmetic processing executed in the super-resolution unit 50, 50a, 50b, 50c, or 50d in FIG. 4, FIG. 17, FIG. 21, FIG. 24, or FIG. It is also possible. Of course, it is possible to form the super-resolution section 50, 50a, 50b, 50c or 50d only by hardware. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 5]
The functions of the super-resolution unit 50, 50a, 50b, 50c, or 50d can be realized by an external device (for example, a personal computer; not shown) different from the imaging device 1. In this case, a super-resolution unit equivalent to the super-resolution unit 50, 50a, 50b, 50c, or 50d is provided in the external device, and a plurality of low-resolution images are acquired by the imaging device 1, and then the plurality of The image information of the low-resolution image may be supplied to the external device wirelessly or by wire or via a recording medium.

[Note 6]
For example, it can be considered as follows. The super-resolution unit 50, 50a, 50b, 50c, or 50d can also be called an image processing apparatus. The super-resolution processing unit 57 functions as a high-resolution image generation unit. It can be considered that the generation unit 55 is included in the high-resolution image generation means.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram of the super-resolution process using a MAP system. It is a flowchart showing the flow of the super-resolution process which concerns on embodiment of this invention. It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 1st Example of this invention. It is the figure which showed the flow of operation | movement of each site | part of FIG. 4 in time series. It is an internal block diagram of the blur amount estimation part which concerns on 1st Example of this invention. It is a figure showing the Laplacian filter as an example of the high pass filter (HPF) of FIG. FIG. 5 is an internal block diagram of a super-resolution processing unit in FIG. 4. It is a figure explaining the experiment content concerning 1st Example of this invention, Comprising: It is a figure which shows the image (a) with large blur amount used in experiment, and the image (b) with small blur amount. It is a figure which shows the high region extraction image of each image of Fig.9 (a) and (b). It is a figure which shows the relationship between the repetition frequency of super-resolution calculation processing, and PSNR of the high resolution image obtained by experiment, and an ideal high resolution image in connection with experiment of 1st Example of this invention. 6 is a flowchart illustrating the flow of super-resolution processing according to the first embodiment of the present invention, with special attention paid to the read / write timing of image data. It is a figure which shows the mode of the magnitude | size change of the magnitude | size of position shift amount concerning 2nd Example of this invention. It is a figure which shows the mode of the magnitude | size change of the magnitude | size of position shift amount concerning 2nd Example of this invention. It is a figure for demonstrating the blurring amount estimation method which concerns on 3rd Example of this invention, Comprising: It is a figure which shows an image (a) with small blur, and an image (b) with large blur. It is a figure which shows the brightness | luminance histogram of each image shown by Fig.15 (a) and (b). It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 4th Example of this invention. It is an internal block diagram of the sensor part provided in the imaging device which concerns on 4th Example of this invention. It is a figure showing the magnitude | size of the blur of an imaging device detected by the sensor part of FIG. It is the figure which specified that image information contains image data and sensor detection data. It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 5th Example of this invention. It is a figure which shows the 1st example of the three observation low-resolution images which concern on 5th Example of this invention. It is a figure which shows the 2nd example of the three observation low-resolution images which concern on 5th Example of this invention. It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 6th Example of this invention. It is an internal block diagram of the imaging part of FIG. FIG. 2 is an internal block diagram of an AF evaluation unit provided in the video signal processing unit of FIG. 1. FIG. 27 is a diagram illustrating an AF evaluation area defined in a frame image by an AF evaluation unit in FIG. 26 and a weighting coefficient set when calculating an AF evaluation value. It is an internal block diagram of the super-resolution part which bears the super-resolution process based on 7th Example of this invention. It is a figure which shows the example of the three observation low-resolution images which concern on 7th Example of this invention. It is a figure which concerns on 8th Example of this invention and shows the pixel arrangement | sequence of the 1st and 2nd observation low resolution image, and the pixel arrangement | sequence of a high resolution image. It is a figure which shows a mode that the amount of position shifts is calculated with a subpixel precision by parabola approximation in connection with 8th Example of this invention. It is a figure which shows a mode that an initial stage high resolution image is produced | generated from the 1st observation low resolution image concerning 8th Example of this invention. FIG. 20 is a diagram illustrating a state in which a spatial filter is applied to a high resolution image in order to reconstruct a low resolution image from the high resolution image according to the eighth embodiment of the present invention. FIG. 25 is a diagram illustrating an example of a matrix for reconstructing a low resolution image from a high resolution image according to the eighth embodiment of the present invention. It is a figure which concerns on 8th Example of this invention and shows the relationship between the pixel arrangement | sequence of the high resolution image which should be updated, and the pixel arrangement | sequence of the low resolution image used for the update. It relates to the eighth embodiment of the present invention, showing a point spread function in the transposed matrix W ₁ ^T and W ₂ ^T, which is used when calculating the feedback value. It is a figure which shows the transposed matrix of the matrix of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Video signal processing part 23 CPU
50, 50a, 50b, 50c, 50d Super-resolution unit 51 Selection unit 52 Blur amount estimation unit 53, 53a, 53b, 53c, 53d Reference image setting unit 54 Position shift detection unit 55 Initial high-resolution image generation unit 56 Selection unit 57 Super-resolution processing unit 75 Sensor unit 76 Subject size detection unit 77 Face state detection unit 80 AF evaluation unit

Claims (15)

In an image processing apparatus that generates a high-resolution image having a resolution higher than the resolution of the low-resolution image from a plurality of low-resolution images,
A reference image setting means for selecting a reference image from the plurality of low resolution images based on image information of the plurality of low resolution images;
An image processing apparatus that generates the high-resolution image by executing a high-resolution process for increasing the resolution with respect to the plurality of low-resolution images on the basis of the reference image. A blur amount estimating means for estimating the size of blur included in each low-resolution image based on the image information;
The image processing apparatus according to claim 1, wherein the reference image setting unit selects the reference image based on an estimation result of the blur amount estimation unit. The image processing apparatus according to claim 2, wherein the blur amount estimation unit estimates a blur size included in each low resolution image based on an amount of a high frequency component in each low resolution image. . The plurality of low-resolution images form a low-resolution image sequence arranged in time series obtained by sequential shooting by an imaging device,
The blur amount estimation means detects a blurring state of the imaging device during a shooting period of the low resolution image sequence based on a positional deviation amount between temporally adjacent low resolution images in the low resolution image sequence. The image processing apparatus according to claim 2, wherein a size relationship between the blur sizes between the plurality of low-resolution images is estimated from a detection result of the blur state. The reference image setting means selects a low resolution image having the smallest estimated blur size as the reference image from the plurality of low resolution images. An image processing apparatus according to claim 1. Each low-resolution image is obtained by shooting with an imaging device,
The image information includes detection data of a sensor that detects a blurring state of the imaging device during an exposure period of each low-resolution image,
The image processing apparatus according to claim 1, wherein the reference image setting unit selects the reference image based on detection data of the sensor. The reference image setting unit selects, as the reference image, a low-resolution image having the smallest blurring of the imaging device during an exposure period among the plurality of low-resolution images. The image processing apparatus described. Subject size detection means for detecting the size of the specific subject included in each low-resolution image on the image based on the image information,
The image processing apparatus according to claim 1, wherein the reference image setting unit selects the reference image based on the detected size of the specific subject. A face state detecting means for detecting a face of the person from each low-resolution image based on the image information and detecting the face state;
The image processing apparatus according to claim 1, wherein the reference image setting unit selects the reference image based on a detection result of the face state detection unit. The standard image setting means sets a low resolution image other than the standard image among the plurality of low resolution images as a reference image,
The image processing apparatus
A displacement detection means for obtaining a displacement amount between the reference image and the reference image with a resolution higher than a pixel interval of the reference image;
A high-resolution image generation unit configured to generate the high-resolution image by performing the high-resolution processing using the obtained positional deviation amount and the plurality of low-resolution images as a reference, based on the reference image; The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Imaging means for acquiring a plurality of low resolution images arranged in time series by sequential shooting;
An image processing apparatus comprising: the image processing apparatus according to claim 1, wherein the image processing apparatus generates a high resolution image by performing high resolution processing on the plurality of low resolution images. An image pickup means having an optical system for forming an optical image corresponding to the image pickup element and the subject on the image pickup element and obtaining a shot image by shooting, and a video signal of a focus evaluation area provided in the shot image A focus evaluation value deriving means for deriving a focus evaluation value based on the imaging evaluation device, and performing an autofocus control by driving the optical system based on the focus evaluation value.
An image processing device that generates a high-resolution image having a resolution higher than the resolution of the low-resolution image from the plurality of low-resolution images by referring to the plurality of captured images obtained from the imaging unit as a plurality of low-resolution images. In addition,
The image processing apparatus includes reference image setting means for selecting a reference image from the plurality of low resolution images based on the focus evaluation value derived for each low resolution image, and the reference image is An imaging apparatus characterized by generating the high-resolution image by executing, on a basis, a resolution enhancement process for increasing the resolution for the plurality of low-resolution images. In the autofocus control, the optical system is driven and controlled so that the focus evaluation value takes a maximum value,
13. The imaging apparatus according to claim 12, wherein the reference image setting unit selects a low resolution image corresponding to a maximum focus evaluation value among the plurality of low resolution images as the reference image. The standard image setting means sets a low resolution image other than the standard image among the plurality of low resolution images as a reference image,
The image processing apparatus includes:
A displacement detection means for obtaining a displacement amount between the reference image and the reference image with a resolution higher than a pixel interval of the reference image;
A high-resolution image generation unit configured to generate the high-resolution image by performing the high-resolution processing using the obtained positional deviation amount and the plurality of low-resolution images as a reference, based on the reference image; The imaging device according to claim 12 or 13, In an image processing method for generating a high-resolution image having a resolution higher than the resolution of the low-resolution image from a plurality of low-resolution images,
Selecting a reference image from the plurality of low resolution images based on image information of the plurality of low resolution images;
An image processing method comprising: generating a high-resolution image by executing, on the plurality of low-resolution images, a high-resolution process that increases resolution with reference to the reference image.