JP2010068093A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To select suitable low resolution images for improvement in image quality of a high-resolution image. <P>SOLUTION: Mutually different low-resolution images F<SB>1</SB>to F<SB>4</SB>have position shifts corrected based upon movement amounts of the low-resolution images F<SB>1</SB>to F<SB>4</SB>, and pixels of the low-resolution images F<SB>1</SB>to F<SB>4</SB>after the position shift correction are arranged on a common coordinate system. Two low-resolution images are selected out of the four low-resolution images. Assuming<SB>4</SB>C<SB>2</SB>(=6) combinations each obtained by selecting two of the four low-resolution images, a distance between pixel positions of two low-resolution images is obtained for each combination. A pixel position distance which is close to a half of a distance between adjacent pixels of a low-resolution image is specified among six distances between pixel positions obtained for first to sixth combinations, and two low-resolution images corresponding to the specified distance between pixel positions are selected. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing on an image. The present invention also relates to an electronic apparatus having an image processing apparatus.

  High resolution processing (super-resolution processing) for generating one high resolution image from a plurality of low resolution images has been proposed. Among the super-resolution processes, a reconfiguration-type super-resolution process using repetitive calculations (repetitive calculations) is known as a typical process. The reconfiguration-type super-resolution processing method using iterative computation is the most effective method among the super-resolution processing methods at present, but it is an optimization computation method that requires many iterative computations. It takes a relatively long time.

  As a method for reducing the processing time, there is a method of selecting m low-resolution images to be used for super-resolution processing from n low-resolution images input to the image processing apparatus (n> m). ). This is because if the number of images used for the super-resolution processing is reduced through the selection processing, the calculation load of the super-resolution processing is reduced and the processing time is reduced.

  In general, when such a selection is performed, after determining one reference frame from the moving images, n images including the reference frame and (n-1) non-reference frames are acquired. On the other hand, a motion amount representing a positional deviation between the reference frame and the non-reference frame is detected, and m images including the reference frame are selected from n images including the reference frame based on the detected motion amount. select.

  For example, in Patent Document 1 described below, after determining one reference frame from a moving image, n frames including a reference frame and (n−1) investigation target frames are acquired. Then, a survey target frame having a relatively small amount of positional deviation (motion amount) from the reference frame is selected, and a high resolution image is generated based on the reference frame and the selected survey target frame.

JP 2006-41603 A

  The m images to be selected are desirably images that are optimal for improving the image quality of high-resolution images. However, in the method as described above, the reference frame is set before the selection process, and the reference frame is always included in the m images to be selected. This can hinder the selection of the optimal image.

  For example, when three images are selected from the first to fourth images, it is assumed that the first image is set as the reference frame. In this case, even if the selection of the second to fourth images is optimal for improving the image quality of the high resolution image, such selection is possible as long as the first image is set as the reference frame. Is not possible. That is, the three images to be selected cannot be obtained as the first, second and third images, the first, second and fourth images, or the first, third and fourth images.

  Note that in the selection process based only on the positional relationship between the amounts of misalignment as in the technique of Patent Document 1, an image that is optimal for substantial resolution improvement (image quality improvement) of a high-resolution image is not always selected. Note that what kind of image is most suitable for substantial resolution improvement of a high-resolution image will be apparent from the description of embodiments of the present invention described later.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an electronic apparatus that generate a high-quality high-resolution image from a plurality of low-resolution images with a low processing load.

An image processing apparatus according to the present invention includes an image input unit that receives n low-resolution images (n is an integer of 3 or more), and a motion amount between different low-resolution images included in the n low-resolution images. A motion amount detection unit to detect, and an image selection unit that selects m of the n low resolution images as m calculation target images based on the detection result of the motion amount detection unit (m is 2 or more) A high resolution image having a resolution higher than the resolution of the low resolution image based on the m calculation target images and the detection result of the motion amount detection unit. A high-resolution processing unit for generating the image processing unit, wherein the image selection unit is configured for each of _n C _m combinations obtained by taking m images from the n low-resolution images. An evaluation is performed based on the detection result of the motion amount detection unit. The m calculation target images are selected by selecting one combination from the evaluation results.

  This makes it possible to select a low-resolution image that is optimal for improving the image quality of a high-resolution image. In addition, since the number of low-resolution images used for computation for generating a high-resolution image is reduced by selection, the processing load can be reduced.

  Specifically, for example, when m low-resolution images for each combination are called m evaluation images, the image selection unit determines each evaluation image based on the detected motion amount between the m evaluation images. Are evaluated in the common coordinate system, and the pixel positional relationship between the m evaluation images is evaluated for each combination. Select a calculation target image.

  More specifically, for example, the m evaluation images include first and second evaluation images, and in each combination, the target pixel position on the second evaluation image is on the second evaluation image. In the pixel position group, the pixel position arranged in the nearest vicinity of the target pixel position on the first evaluation image, and the image selection unit includes the target pixel position on the first evaluation image and the second A distance deriving process for deriving a distance between the target pixel position on the evaluation image is performed for each combination, and among the distances derived for each combination, the distance between adjacent pixels of the low resolution image The distance closest to the predetermined distance based on the above is determined, and the m evaluation images corresponding to the closest distance are selected as the m calculation target images.

  Alternatively, for example, m is 3 or more, and the m evaluation images include first to m-th evaluation images, and in each combination, the target pixel position on the i-th evaluation image is the i-th evaluation image. It is a pixel position arranged in the nearest vicinity of the target pixel position on the first evaluation image in the pixel position group on the image (i is an integer of 2 or more and m or less), and the image selection unit An area deriving process for deriving an area of a polygon formed by connecting m target pixel positions for one evaluation image is executed for each combination, and the size of the area derived for each combination is determined. The m calculation target images are selected by selecting one combination based on the relationship.

  Alternatively, for example, m is 3 or more, the m evaluation images are composed of the first to m-th evaluation images, and in each combination, the target pixel position on the i-th evaluation image is the i-th evaluation image. In the upper pixel position group, the pixel position is arranged closest to the target pixel position on the first evaluation image (i is an integer not smaller than 2 and not larger than m). A plurality of sets of two target pixel positions included in the m target pixel positions for the evaluation image of, and a distance between the two target pixel positions is obtained for each set, and a sum of the obtained distances is obtained. The derived total sum derivation process is executed for each combination, and the m calculation target images are selected by selecting one combination based on the magnitude relation of the sum derived for each combination.

  An electronic apparatus according to the present invention is an electronic apparatus that includes the image processing apparatus, acquires n images by photographing or input from the outside, and supplies image signals of the n images to the image processing apparatus. is there. The image processing apparatus receives the n images as the n low-resolution images.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the image processing apparatus and electronic device which produce | generate a high-resolution high-resolution image from several low-resolution images with low processing load.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to tenth embodiments will be described later. First, matters common to each embodiment or items referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera, for example. The imaging device 1 can capture a moving image and a still image, and can also capture a still image simultaneously during moving image capturing.

[Description of basic configuration]
The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and a DRAM (Dynamic Random Access Memory). An internal memory 17 such as an SD (Secure Digital) card or a magnetic disk, an expansion processing unit 19, a VRAM (Video Random Access Memory) 20, an audio output circuit 21, and a TG (timing generator). 22, a CPU (Central Processing Unit) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25.

  The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and provides the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the internal memory 17 during signal processing as necessary.

  The imaging unit 11 includes an imaging system (image sensor) 33, an optical system (not shown), a diaphragm, and a driver. Incident light from the subject enters the image sensor 33 via the optical system and the stop. Each lens constituting the optical system forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

  The image sensor 33 is a solid-state image sensor composed of a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 33 photoelectrically converts an optical image incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels (not shown in FIG. 1) that are two-dimensionally arranged in a matrix, and in each photographing, each light receiving pixel has a charge amount signal corresponding to the exposure time. Stores charge. The electrical signal from each light receiving pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to the subsequent AFE 12 in accordance with the drive pulse from the TG 22.

  The AFE 12 amplifies the analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The degree of amplification of signal amplification in the AFE 12 is controlled by the CPU 23. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The video signal is generally composed of a luminance signal Y representing the luminance of the image and color difference signals U and V representing the color of the image.

  The microphone 14 converts the peripheral sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. The compressed video signal is recorded in the external memory 18 at the time of capturing and recording a moving image or a still image. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting and recording, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being correlated with each other in time by the compression processing unit 16, and after compression, Recorded in the external memory 18.

  The recording button 26a is a push button switch for instructing start / end of moving image shooting and recording, and the shutter button 26b is a push button switch for instructing shooting and recording of a still image.

  The operation mode of the imaging apparatus 1 includes a shooting mode capable of shooting a moving image and a still image, and a playback mode for reproducing and displaying the moving image and the still image stored in the external memory 18 on the display unit 27. . Transition between the modes is performed according to the operation on the operation key 26c.

  In the shooting mode, shooting is sequentially performed at a predetermined frame period, and a shot image sequence is acquired from the image sensor 33. An image sequence typified by a captured image sequence refers to a collection of images arranged in time series. Data representing an image is called image data. Image data can also be considered as a kind of video signal. One image is represented by image data for one frame period. One image represented by image data for one frame period is also called a frame image.

  When the user presses the recording button 26a in the shooting mode, under the control of the CPU 23, the video signal obtained after the pressing and the corresponding audio signal are sequentially recorded in the external memory 18 via the compression processing unit 16. . When the user presses the recording button 26a again after starting the moving image shooting, the recording of the video signal and the audio signal to the external memory 18 is completed, and the shooting of one moving image is completed. In the shooting mode, when the user presses the shutter button 26b, a still image is shot and recorded.

  In the playback mode, when the user performs a predetermined operation on the operation key 26c, a compressed video signal representing a moving image or a still image recorded in the external memory 18 is expanded by the expansion processing unit 19 and written to the VRAM 20. . In the shooting mode, the video signal is normally generated by the video signal processing 13 regardless of the operation contents of the recording button 26a and the shutter button 26b, and the video signal is written in the VRAM 20.

  The display unit 27 is a display device such as a liquid crystal display, and displays an image corresponding to the video signal written in the VRAM 20. In addition, when a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded in the external memory 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

  The video signal processing unit 13 is configured to be able to perform super-resolution processing in cooperation with the CPU 23. Through the super-resolution processing, one high-resolution image is generated from a plurality of low-resolution images. The video signal of the high resolution image can be recorded in the external memory 18 via the compression processing unit 16. The resolution of the high resolution image is higher than that of the low resolution image, and the number of pixels in the horizontal and vertical directions of the high resolution image is larger than that of the low resolution image. For example, when an instruction to capture a still image is given, a plurality of frame images as a plurality of low resolution images are acquired, and a high resolution image is generated by performing super-resolution processing on them. Alternatively, for example, the super-resolution processing is performed on a plurality of frame images as a plurality of low resolution images obtained at the time of moving image shooting.

[Basic concept of super-resolution processing]
The basic concept of super-resolution processing will be briefly described. There is a method called a reconstruction type as a type of super-resolution processing. On the other hand, there is a method for realizing super-resolution processing by repetitive calculation (repetitive calculation algorithm), and this repetitive calculation can also be applied to reconstruction-type super-resolution processing. In the present embodiment, a main example is a reconstruction type super-resolution process based on a MAP (Maximum A Posterior) method using a repetitive calculation, which is a kind of super-resolution process that can be realized by a repetitive calculation.

  FIG. 2 shows a conceptual diagram of a reconstruction type super-resolution process based on the MAP method using repetitive operations. In this super-resolution processing, one high-resolution image is estimated from a plurality of low-resolution images obtained by actual shooting, and the original plurality of low-resolution images are estimated by degrading the estimated high-resolution image. To do. A low resolution image obtained by actual photographing is particularly called an “real low resolution image”, and an estimated low resolution image is particularly called an “estimated low resolution image”. Thereafter, the high resolution image and the low resolution image are repeatedly estimated so that the error between the actual low resolution image and the estimated low resolution image is minimized, and the finally acquired high resolution image is output.

  FIG. 3 is a flowchart showing the flow of super-resolution processing corresponding to FIG. First, in step S11, an initial high resolution image is generated from an actual low resolution image. In subsequent step S12, the original actual low resolution image for constructing the current high resolution image is estimated. The estimated image is referred to as an estimated low resolution image as described above. In the subsequent step S13, an update amount for the current high resolution image is derived based on the difference (difference image) between the actual low resolution image and the estimated low resolution image. This update amount is derived so that the error between the actual low-resolution image and the estimated low-resolution image is minimized by repeatedly executing the processes in steps S12 to S14. In the subsequent step S14, the current high resolution image is updated using the updated amount, and a new high resolution image is generated. Thereafter, the process returns to step S12, and the newly generated high resolution image is regarded as the current high resolution image, and the processes of steps S12 to S14 are repeatedly executed. Basically, as the number of repetitions of each of the processes in steps S12 to S14 increases, the resolution of the obtained high resolution image is substantially improved (resolution is improved), and an ideal high resolution image can be obtained.

  Super-resolution processing based on the above-described operation flow is performed in the imaging apparatus 1. In addition to super-resolution processing based on the MAP method, it is also possible to use super-resolution processing based on the ML (Maximum-Likelihood) method, the POCS (Projection Onto Convex Set) method, or the IBP (Iterative Back Projection) method. is there.

  In reconstruction-type super-resolution processing represented by the MAP method, after generating an initial high-resolution image, calculation processing (super-resolution calculation processing) including calculation of an update amount and update of a high-resolution image based on the update amount Are repeatedly executed, but it is not essential to repeatedly execute this arithmetic processing. That is, an initial high resolution image may be generated from a plurality of low resolution images, and the initial high resolution image may be handled as a high resolution image to be finally obtained without being updated.

[Low-resolution image selection guideline: Selection optimization process]
When performing super-resolution processing, multiple real low-resolution images used for high-resolution image estimation are selected from the frame image sequence, and the selected real low-resolution images are suitable for high-resolution image estimation. Otherwise, it is difficult to generate a good high-resolution image having high resolution. The video signal processing unit 13 has a function of selecting an actual low resolution image that is optimal for generating a good high resolution image. Processing for realizing this function is hereinafter referred to as “selection optimization processing”, and each of the low resolution images selected by the selection optimization processing is referred to as “calculation target image”.

  By this selection optimization process, a good high-resolution image can be generated. Also, making this selection corresponds to limiting the number of actual low resolution images used to generate a high resolution image. Therefore, by executing this selection, the calculation load of the super-resolution processing in the video signal processing unit 13 is reduced, and the acquisition speed of the high-resolution image is increased.

  The basic selection guideline for the calculation target image by the selection optimization process will be described. In the description of this basic selection guideline, for simplification of explanation, it is assumed that each image is a one-dimensional image, and two of the three actual low resolution images obtained by photographing at different times are displayed. Assume that two calculation target images are selected. Assume that three real low-resolution images are formed from the first to third real low-resolution images, and that the high-resolution image has twice the resolution of the low-resolution image.

It is assumed that the imaging device 1 has moved due to camera shake or the like between the first to third real low-resolution images. If the subject is stationary during the shooting of the first to third actual low resolution images, each of the second and third actual low resolution images is an image obtained by shifting the position of the first actual low resolution image. Can be considered. The positional deviation amounts of the second and third actual low resolution images with reference to the first actual low resolution image are represented by D ₁₂ and D _13, respectively.

FIG. 4 is a diagram showing pixels on the first to third real low-resolution images arranged in a common coordinate system. However, at the time of arrangement in the common coordinate system, the second and third real low-resolution images are referenced based on the pixels of the first real low-resolution image by an amount corresponding to the displacements D ₁₂ and D ₁₃ . The pixels are shifted and arranged. In FIG. 4, black triangles, black squares, and black circles indicate the pixels (and pixel positions) of the first to third real low-resolution images on the common coordinate system, respectively. The position where the pixel is arranged is called a pixel position. In FIG. 4, pixels of the high-resolution image arranged with reference to the pixel positions of the first low-resolution image are also indicated by white circles. Also, the adjacent pixel interval of the low resolution image is represented by pp _L. Since it is assumed that the high resolution image has twice the resolution of the low resolution image, the adjacent pixel interval of the high resolution image is pp _L / 2.

A curve 301 in FIG. 4 represents an analog luminance signal on the imaging surface of the imaging element 33 when the first actual low-resolution image is captured. This analog luminance signal is a signal corresponding to the luminance distribution of the subject of the imaging apparatus 1. The imaging element 33 samples the analog luminance signal 301 at the sampling interval pp _L , thereby obtaining the luminance value of each pixel of the actual low resolution image.

When the pixel position of the high resolution image is determined with reference to the pixel position of the first actual low resolution image, the pixel position of the adjacent pixels 311 and 312 on the first actual low resolution image is the same as the pixel position. Two pixels 321 and 322 on the high-resolution image are arranged, and a pixel 323 of the high-resolution image that should be called an interpolation pixel is further arranged between the pixels 321 and 322. Although the luminance values of the pixels 321 and 322 can be generated from the luminance values of the pixels 311 and 312, the luminance value of the pixel 323 can be set to the ideal value (the actual object at the position of the pixel 323 only with the luminance values of the pixels 311 and 312. It is difficult to approach the (luminance value). Therefore, the second or third real low-resolution image is used to generate the luminance value of the pixel 323. At this time, as shown in FIG. 4, if D ₁₂ <D ₁₃ ≈pp _L / 2, the luminance value of the third actual low-resolution image is greater than the luminance value of the second actual low-resolution image. It has a value closer to the ideal value.

  Therefore, if one of the two calculation target images is determined as the first actual low resolution image, the third actual low resolution image is selected as the other calculation target image. As a result, the pixel value of the interpolated pixel can be acquired based on a value closer to the ideal value (actual subject luminance value), and a substantial resolution improvement in a high-resolution image can be expected.

The following way of thinking can also be made. If D ₁₂ ≈0, then the first and second real low resolution images are substantially the same image for the generation of the high resolution image, and the presence of the second real low resolution image is the high resolution image. Hardly or not at all. Therefore, an actual low-resolution image that is as close to pp _L / 2 as possible with respect to the first actual low-resolution image should be included in the calculation target image.

  The selection optimization process is executed based on the basic selection guideline as described above. In the above example, it is assumed that the first actual low-resolution image is included in the calculation target image. However, if the second actual low-resolution image is determined as one calculation target image, the first actual low-resolution image is determined as the first calculation target image. And the second actual low-resolution image or the second and third actual low-resolution images are selected as the two calculation target images, and the two most ideal real low-resolution images (that is, the first and second actual low-resolution images) The third actual low-resolution image) cannot be selected as two calculation target images. In consideration of this, in the actual selection optimization process, an optimal combination of images to be calculated is searched for in a so-called brute force formula without assuming that any actual low-resolution image is included in the image to be calculated. To do.

  Hereinafter, specific implementation methods of the selection optimization process, contents of the super-resolution process, and / or technical matters related thereto will be described in the first to tenth embodiments. As long as no contradiction arises, the matters described in one embodiment can be applied to other embodiments. In particular, the matters described in the first embodiment are applied to each embodiment other than the first embodiment unless there is a contradiction.

In the following description, a frame image obtained by shooting is treated as an actual low-resolution image unless otherwise specified, and a plurality of arbitrary images represented by a low-resolution image and a high-resolution image are provided in both the horizontal and vertical directions. Is a two-dimensional image in which the pixels are arranged in a matrix. An actual low resolution image (frame image) obtained by photographing at time t _k is represented by F _k . Here, k is a natural number. The time length between the time t _k and the time t _{k + 1} corresponds to, for example, a frame period. In addition, a frame at time t _k is particularly referred to as a k-th frame.

Further, in this specification, for the sake of simplification of description, a name corresponding to the symbol may be abbreviated or omitted by using the symbol. For example, in this specification, the actual low-resolution image F ₁ may be simply expressed as “image F ₁ ” or “F ₁ ”, but the former and the latter indicate the same thing.

<< First Example >>
A first embodiment of the present invention will be described. FIG. 5 shows a block diagram of a part related to the super-resolution processing in the imaging apparatus 1. The video signal processing unit 13 in FIG. 5 includes each part referred to by reference numerals 41 to 49.

The frame memory 41 temporarily stores image data of an actual low resolution image for one frame represented by the digital signal from the AFE 12. The frame memory 42 temporarily stores image data of an actual low resolution image for one frame stored in the frame memory 41. The frame memory 43 temporarily stores image data of an actual low resolution image for one frame stored in the frame memory 42. The frame memory 44 temporarily stores image data of an actual low resolution image for one frame stored in the frame memory 43. The contents stored in the frame memory 41 are sequentially transferred to the frame memories 42, 43 and 44 every time one frame elapses. Thereby, at the end of the fourth frame, the image data of the actual low resolution images F ₁ , F ₂ , F ₃ and F ₄ are recorded in the frame memories 44, 43, 42 and 41, respectively.

The motion amount calculation unit 45 is supplied with the image data of the actual low resolution image of the current frame from the AFE 12 and the image data of the actual low resolution image of the previous frame from the frame memory 41. The motion amount calculation unit 45 calculates a motion amount representing the amount of positional deviation between the two given real low-resolution images by comparing the two given image data. This motion amount is a two-dimensional amount including a horizontal component and a vertical component, and is expressed as a so-called motion vector. As the current frame changes to the second, third, and fourth frames, as shown in FIG. 6, in the motion calculation unit 45, the motion amount MV ₁₂ between the images F ₁ and F ₂ and the image F ₂ are displayed. and a motion amount MV ₂₃ for between F _3, the motion amount MV ₃₄ for between images F ₃ and F _4, is obtained. Each obtained motion amount is stored in the motion amount storage unit 46. Further, from the motion amounts MV ₁₂ , MV ₂₃ and MV ₃₄ , the motion amount MV ₁₃ between the images F ₁ and F ₃ , the motion amount MV ₂₄ between the images F ₂ and F _{4 and} between the images F ₁ and F ₄ . Motion amount MV _{14 is} also calculated, and these are also stored in the motion amount storage unit 46.

The motion amount calculation unit 45 calculates a motion amount between two actual low-resolution images using a representative point matching method, a block matching method, a gradient method, or the like. The motion amount calculated here has a so-called sub-pixel resolution having a resolution higher than the pixel interval of the actual low-resolution image. That is, the amount of motion is calculated with a distance shorter than the interval pp _L between two pixels adjacent in the horizontal or vertical direction in the actual low-resolution image as a minimum unit. A known calculation method can be used as a method for calculating a positional deviation amount having sub-pixel resolution. For example, the method described in Japanese Patent Application Laid-Open No. 11-345315 and the method described in “Okutomi,“ Digital Image Processing ”, Second Edition, CG-ARTS Association, issued on March 1, 2007” (p. 31). 205).

  Image data of n actual low-resolution images is given to the image selection unit 47 from the memory 40 including the frame memories 41 to 44. Then, based on each motion amount stored in the motion amount storage unit 46, m actual low resolution images are selected from the n actual low resolution images by the selection optimization process, and the selected m actual low resolution images are selected. The resolution image is output to the super-resolution processing unit 48 as m calculation target images. In the example shown in FIG. 5, n = 4, but n is an arbitrary integer of 3 or more. In the case of n ≠ 4, the number of frame memories forming the memory 40 may be appropriately modified from that shown in FIG. m is an arbitrary integer of 2 or more that satisfies the inequality “n> m”.

The super-resolution processing unit 48 receives a motion amount between different calculation target images included in the m calculation target images from the motion amount storage unit 46, and calculates the motion amount and m images from the image selection unit 47. One high-resolution image is generated by super-resolution processing using the target image (m actual low-resolution images). As a super-resolution processing method executed by the super-resolution processing unit 48, any method including the above-described method can be adopted. The high resolution image is generated so as to have twice the resolution of the low resolution image, for example. In this case, the interval between adjacent pixels along the horizontal or vertical direction on the high-resolution image is pp _L / 2.

  The signal processing unit 49 generates a video signal (luminance signal and color difference signal) of the high resolution image based on the image data of the high resolution image generated by the super-resolution processing unit 48. When the super-resolution processing is not performed, the signal processing unit 49 can generate a video signal based on the image data of the actual low-resolution image output from the AFE 12. The video signal generated by the signal processing unit 49 can be stored in the external memory 18 through the compression processing by the compression processing unit 16.

  A method of selecting the calculation target image by the selection optimization process of the image selection unit 47 will be described in detail. In the first embodiment, it is assumed that n = 4 and m = 2.

The n actual low-resolution images F _{1 to} F _n input from the memory 40 to the image selection unit 47 are considered to be arranged on a two-dimensional common coordinate system (hereinafter referred to as a common coordinate system). FIG. 7 shows a grid and pixels of each real low-resolution image arranged in the common coordinate system. Code 401 is affixed black triangle indicates the position at which the pixel image F ₁ with representative of the pixels of the image F ₁ are disposed, the image F ₂ with black code 402 is attached rectangle represents a pixel of the image F ₂ indicates the position at which the pixels are arranged, a black circle sign 403 is attached represents a position where a pixel of the image F ₃ are arranged with representing a pixel of the image F _3, star sign 404 is attached type shapes (ten square) represents a position where a pixel of the image F ₄ are arranged with representing a pixel of the image F _4. In FIG. 7 (and FIG. 8 and the like described later), reference numeral 401 is given only to a part of the triangle group representing the pixel group forming the image F ₁ . The same applies to reference numerals 402 to 404.

The pixels on the image F _k are arranged in a grid pattern at equal intervals in the horizontal and vertical directions of the common coordinate system (k = 1, 2, 3, 4). A grid in which the pixels forming the image F _k are two-dimensionally arranged in the common coordinate system is called a grid of the image F _k .

The image F ₂ can be regarded as an image obtained by shifting the position of the image F ₁ by an amount corresponding to the movement amount MV ₁₂ . The image selection unit 47 sets the grids of the images F ₁ and F ₂ on the common coordinate system so that the positional deviation of the image F ₂ with respect to the image F ₁ is cancelled. For example, when the size of the movement amount of the image F ₂ as viewed from the image F ₁ is and a right direction the movement amount is pp _L / 2, the grid of the image F _1, pp leftward _L The grid of the image F ₂ is set at a position shifted by / 2. Similarly, the grids of the images F ₃ and F ₄ are set on the common coordinate system with the grid of the image F ₁ as a reference. When considering the grid of the image F ₁ as a reference, the grid positions of the images F _{1 to} F ₄ are determined based on the motion amounts MV ₁₂ , MV _13, and MV ₁₄ . Note that the grid positions of the images F _{1 to} F ₄ on the common coordinate system may be set on the basis of the image F ₂ , F _3, or F ₄ .

Image selecting section 47 to select two images (operation target image) from among the four images F ₁ to F _4, the two images included in the image F ₁ to F ₄ of the pixel position Calculate the distance. This calculation is performed by the number of combinations for selecting two images from the _four images F _{1 to} F ₄ . The number of combinations that can be taken from n to m is _n C _m . In this example, since n = 4 and m = 2 are assumed, _n C _m = ₄ C ₂ = 6, and a combination of selecting two images from _four images F _{1 to} F ₄ is used. There are first to sixth combinations.

Each of the two images for calculation of the distance between the pixel positions is called an evaluation image, and the two evaluation images are represented by symbols F _A and F _B. Then, in the first, second, third, fourth, fifth, and sixth combinations, (A, B) = (1,2), (1,3), (1,4), ( 2, 3), (2, 4) and (3,4). The evaluation image is any real low-resolution image arranged in the common coordinate system based on the amount of motion (the same applies to other examples described later).

A method for calculating the distance between pixel positions for the first combination will be described. The distance between pixel positions is a distance on the common coordinate system. When calculating the inter-pixel position distance for the first combination, the images F ₁ and F ₂ are treated as the evaluation images F _A and F _B from (A, B) = (1, 2), and attention is paid to the image F ₁ . A distance d ₁₂ on the common coordinate system between the pixel position and the target pixel position of the image F ₂ is calculated.

In time of calculating the distance d _12, the target pixel position of the image F ₁ is any one pixel location on the image F _1. Position of the black triangle symbols 401a is attached in FIG. 8, a case was the target pixel position of the image F _1.

The image selection unit 47 sets a reference rectangular area centered on the target pixel position of the evaluation image F _A. The reference rectangular area is a square area, and the length of one side of the square is pp _L. When considering the first combination, since the evaluation image F _A is the image F ₁ and the target pixel position of the evaluation image F _A is the pixel position 401a, one side centered on the pixel position 401a shown in FIG. A square region 410 having a length of pp _L is set as the reference rectangular region.

Then, when calculating the pixel position distance between the first combination, the target pixel position of the image F ₂ is located at the reference rectangular region 410 is a pixel position 402a of the image F _2. Here, a pixel position located on the boundary line between the inside and outside of the reference rectangular area 410 is also treated as a pixel position located in the reference rectangular area 410 (the same applies to the reference rectangular areas other than the reference rectangular area 410). Pixel position 402a, of the pixel position group image F _2, a pixel position that is located closest to the target pixel position 401a of the image F _1. The distance d ₁₂ corresponding to the pixel position distance between the first combination is the distance between the pixel position 401a and the pixel position 402a.

The distance between pixel positions for the second and third combinations is calculated in the same manner. The distance d ₁₃ corresponding to the inter-pixel position distance for the second combination is the distance between the pixel position 401 a of the image F _{1 and} the pixel position 403 a of the image F ₃ located in the reference rectangular area 410. The distance d ₁₄ corresponding to the inter-pixel position distance for the third combination is the distance between the pixel position 401 a of the image F ₁ and the pixel position 404 a of the image F ₄ located in the reference rectangular area 410. is there.

When calculating the inter-pixel position distance for the fourth combination, the images F ₂ and F ₃ are treated as the evaluation images F _A and F _B from (A, B) = (2, 3), and the fifth combination When calculating the distance between pixel positions for, the images F ₂ and F ₄ are treated as the evaluation images F _A and F _B from (A, B) = (2, 4). Therefore, as shown in FIG. 9, around the pixel-of-interest position 402b is any one pixel location on the image F _2, reference rectangular region 420 is set. This reference rectangular area 420 is also a square area having a side length of pp _L as described above.

Then relates a combination of the fourth and fifth, the target pixel position of the evaluation image F _B, respectively, located in the reference rectangular region 420 is a pixel position 404b of the pixel position 403b and the image F ₄ image F ₃ . Therefore, the distance d ₂₃ corresponding to the distance between pixel positions for the fourth combination is the distance between the pixel position 402b and the pixel position 403b, and the distance corresponding to the distance between the pixel positions for the fifth combination. d ₂₄ is a distance between the pixel position 402b and the pixel position 404b.

A distance d ₃₄ corresponding to the inter-pixel position distance for the sixth combination is calculated in the same manner. The distance between the pixel positions 403b and 404b shown in FIG. 9, if it is the reference distance d _REF less, the distance between pixel locations 403b and 404b are the distance d _34. Here, the reference distance d _REF corresponds to half the length of the diagonal line of the reference rectangular area. That is, the reference distance d _REF is obtained by multiplying pp _L / 2 by the square root of 2.

As described above, the image selection unit 47 executes the distance derivation process for obtaining the distance between the target pixel position of the evaluation image F _{A and} the target pixel position of the evaluation image F _B for each of the first to sixth combinations. Thus, the six distances d ₁₂ , d ₁₃ , d ₁₄ , d ₂₃ , d ₂₄ and d ₃₄ are obtained. Then, the maximum distance among the six distances is specified. Since the distance obtained by the distance deriving process is always less than or equal to the reference distance d _REF , the maximum distance specified here is the distance closest to the reference distance d _REF among the six distances. Then, the image selection unit 47 selects two actual low resolution images corresponding to the specified distance as two calculation target images. In the example shown in FIGS. 8 and 9, since the distance d ₂₄ is the largest of the six distances, the actual low resolution images F ₂ and F ₄ corresponding to the distance d ₂₄ are selected as two calculation target images. .

After fixedly setting any one of the _four images F _{1 to} F ₄ as a reference image, 2 including the reference image based on the relationship between the pixel position of the reference image and the pixel position of another image. A method of selecting one calculation target image is also conceivable. When this method is employed, if the reference image is the image F ₂ or F ₄ in the example shown in FIGS. 8 and 9, the images F ₂ and F ₄ may be selected as two calculation target images. Although it is possible, if the reference image is the image F ₁ or F ₃ , the images F ₂ and F ₄ cannot be selected as two calculation target images. As described above, in the method of selecting m calculation target images including the reference image after determining the reference image, the optimal calculation target image cannot be selected depending on how the reference image is set.

  On the other hand, in the selection optimization process described above, it is possible to select m calculation target images that are optimal for super-resolution processing from n actual low-resolution images without defining a reference image. That is, as described in the description of the basic selection guideline for the calculation target image, it becomes possible to select the calculation target image that is most suitable for the substantial resolution improvement of the high resolution image, and the substantial resolution (resolution) of the high resolution image can be selected. ) Is maximized.

<< Second Example >>
A second embodiment of the present invention will be described. The video signal processing unit 13 according to the second embodiment is the same as that shown in FIG. 5, and the items described in the first embodiment are also applied to the second embodiment. However, in the second embodiment, m = 3. Although n may be any number as long as it is 4 or more, it is assumed that n = 4 for the sake of concrete description.

In this case, the image selection unit 47 selects three actual low resolutions from among the _four actual low resolution images F _{1 to} F ₄ through the selection optimization process based on each motion amount stored in the motion amount storage unit 46. An image is selected, and the selected three real low-resolution images are output to the super-resolution processor 48 as three calculation target images.

In the second embodiment, in order to select the three images (operation target image) from among the four images F ₁ to F _4, evaluation of three, three images included in the image F ₁ to F ₄ Extracted as images, the suitability for super-resolution processing is evaluated for the three evaluation images. The number of combinations that can be taken from n to m is _n C _m . In this example, since n = 4 and m = 3 are assumed, _n C _m = ₄ C ₃ = 4, and a combination of selecting three images from the _four images F _{1 to} F ₄ is used. There are first to fourth combinations.

Three evaluation images are represented by symbols F _A , F _B and F _C. Then, in the first, second, third, and fourth combinations, (A, B, C) = (1, 2, 3), (1, 2, 4), (1, 3, 4), respectively. , (2, 3, 4).

Image selecting unit 47, a triangle area derivation process of deriving the area of a triangle connecting the target pixel position of the evaluation image F evaluation target pixel position in image _A F evaluation image F _C and the position of the pixel of interest _B, first to Three real low-resolution images corresponding to the largest area among the four areas derived for the first to fourth combinations are executed for each of the fourth combinations. Select as an image.

The triangle area derivation process will be described. First, the target pixel position of the evaluation image F _A is an arbitrary one pixel positions on the evaluation image F _A. As in the first embodiment, the image selection unit 47 sets a reference rectangular area centered on the target pixel position of the evaluation image F _A. Its located reference rectangular region, the pixel position of the evaluation images F _B and F _C is the pixel of interest position of the evaluation images F _B and F _C. Thus, the target pixel position of the evaluation image F _B, the evaluation of the pixel position groups of images F _B, a pixel position that is located closest to the target pixel position of the evaluation image F _A, the target pixel position of the evaluation image F _C Is a pixel position located closest to the target pixel position of the evaluation image F _A in the pixel position group of the evaluation image F _C.

Accordingly, in the example shown in FIGS. 8 and 9, the area S _{T1 of the} triangle connecting the pixel positions 401a, 402a, and 403a is calculated for the first combination, and the pixel positions 401a, 402a, and 402 are calculated for the second combination. The area S _{T2 of the} triangle connecting 404a is calculated, the area S _{T3 of the} triangle connecting the pixel positions 401a, 403a, and 404a is calculated for the third combination, and the pixel positions 402b, 403b, and 404 for the fourth combination are calculated. area S _T4 of a triangle connecting the 404b are calculated.

Then, the image selection unit 47 specifies the maximum area among the areas S _{T1 to} S _T4 , and selects three actual low resolution images corresponding to the maximum area as three calculation target images. For example, among the area S _T1 to S _T4, if the area S _T1 is at a maximum, the image F _1, F ₂ and F ₃ corresponding to the area S _T1 is selected as the three calculation target image, the area S _{If T4} is the maximum, the images F ₂ , F ₃ and F ₄ corresponding to the area S _T4 are selected as three calculation target images.

FIG. 10 shows the pixel positional relationship of the images F _{1 to} F ₄ on the common coordinate system when the images F _{1 to} F ₃ are selected as three calculation target images. In the example of FIG. 10, the triangle formed by the target pixel positions of the images F _{1 to} F ₃ is an isosceles triangle having a base and a height of pp _L / 2. When this isosceles triangle is formed, the area of the triangle is maximized. By selecting three real low-resolution images corresponding to the triangle as close as possible to the isosceles triangle (that is, the area as large as possible) as three calculation target images, the substantial resolution of the high-resolution image can be reduced. Maximize.

If the pixel positions of the images F _{1 to} F ₄ on the common coordinate system are in the situation as shown in FIG. 11, even if the images F _{1 to} F ₃ are selected as three calculation target images, the image The presence of F ₂ and F ₃ hardly contributes to a substantial resolution improvement of the high resolution image. From this point, the significance of the area-based selection optimization process is understood.

<< Third Example >>
A third embodiment of the present invention will be described. The video signal processing unit 13 according to the third embodiment is the same as that shown in FIG. 5, and the items described in the first embodiment are also applied to the third embodiment. However, in the third embodiment, m = 4. Accordingly, n is an integer of 5 or more, and 5 or more actual low resolution images are input to the image selection unit 47 via the memory 40. For the sake of concrete explanation, it is assumed that n = 5. In this case, the motion amount calculation unit 45 calculates the motion amount MV ₄₅ between the actual low-resolution images F ₄ and F ₅ in addition to the motion amounts MV ₁₂ , MV _23, and MV ₃₄ . Based on the motion amounts stored in the motion amount storage unit 46, the pixel positional relationship of the images F _{1 to} F ₅ on the common coordinate system is determined.

The image selection unit 47 selects four actual low-resolution images from the _five actual low-resolution images F _{1 to} F ₅ through the selection optimization process based on each motion amount stored in the motion amount storage unit 46. Then, the selected four real low-resolution images are output to the super-resolution processor 48 as four calculation target images.

In the third embodiment, in order to select the five four images from the image F ₁ to F ₅ (calculation target image), the evaluation of four four images included in the image F ₁ to F ₅ The images are extracted and the suitability for the super-resolution processing is evaluated for the four evaluation images. The number of combinations that can be taken from n to m is _n C _m . In this example, since n = 5 and m = 4 are assumed, _n C _m = ₅ C ₄ = 5, and a combination of selecting four images from _five images F _{1 to} F ₅ is used. There are first to fifth combinations.

Four evaluation images are represented by symbols F _A , F _B , F _C and F _D. Then, in the first, second, third, fourth, and fifth combinations, (A, B, C, D) = (1, 2, 3, 4), (1, 2, 3, 5), respectively. ), (1,2,4,5), (1,3,4,5), (2,3,4,5).

Image selecting unit 47 derives the area of the rectangle connecting the target pixel position of the evaluation image F evaluation target pixel position in image _A F evaluation image F _D between the target pixel position and the pixel-of-interest position of the evaluation image F _C of _B The square area derivation process is executed for each of the first to fifth combinations, and the four actual areas corresponding to the maximum area among the five areas derived for the first to fifth combinations are processed. The low resolution image is selected as four calculation target images.

The square area derivation process will be described. First, the target pixel position of the evaluation image F _A is an arbitrary one pixel positions on the evaluation image F _A. As in the first embodiment, the image selection unit 47 sets a reference rectangular area centered on the target pixel position of the evaluation image F _A. Its located reference rectangular region, the evaluation image F _B, the pixel positions of F _C and F _D, is a target pixel position of the evaluation image F _B, F _C and F _D. Thus, the target pixel position of the evaluation image F _B, the evaluation of the pixel position groups of images F _B, a pixel position that is located closest to the target pixel position of the evaluation image F _A, the target pixel position of the evaluation image F _C the evaluation of the pixel position group image F _C, a recently pixel positions located near the target pixel position of the evaluation image F _a, the target pixel position of the evaluation image F _D is the pixel position group evaluation image F _D Among these, the pixel position is located closest to the target pixel position of the evaluation image F _A.

Therefore, if the pixels of the images F _{1 to} F ₄ are arranged at the positions shown in FIG. 8 on the common coordinate system, the pixel positions 401a, 402a, 403a, and 404a are connected to the first combination. A square area S _R1 is calculated. Similarly, square areas S _{R2 to} S _R5 are calculated for the second to fifth combinations, respectively.

The image selection unit 47 specifies the maximum area among the areas S _{R1 to} S _R5 , and selects four actual low-resolution images corresponding to the maximum area as four calculation target images. Maximum example, among the area S _R1 to S _R5, if the area S _R1 is maximum, the image F ₁ to F ₄ corresponding to the area S _R1 is selected as four calculation target image, the area S _R5 is If so, the images F _{2 to} F ₅ corresponding to the area S _R5 are selected as four calculation target images.

FIG. 12 shows the pixel positional relationship of the images F _{1 to} F ₄ on the common coordinate system when the images F _{1 to} F ₄ are selected as four calculation target images. In the example of FIG. 12, the quadrangle formed by the target pixel positions of the images F _{1 to} F ₄ is a square having a side length of pp _L / 2. When this square is formed, the square area is maximized. By selecting four actual low-resolution images corresponding to the square as close as possible to the square (that is, as large as possible) as four calculation target images, the substantial resolution of the high-resolution image is maximized. Is planned.

<< 4th Example >>
A fourth embodiment of the present invention will be described. In the second and third embodiments, assuming that m = 3 and m = 4, the calculation target image is selected through the derivation of the area of the triangle and the quadrangle. In the fourth embodiment, Assuming that m is an arbitrary integer equal to or greater than 3, the selection optimization process is generalized. Since n> m, n is an integer of 4 or more in the fourth embodiment. If n = 4 and m = 3, the selection optimization process according to the fourth embodiment is the same as that of the second embodiment. If n = 5 and m = 4, the selection optimization process according to the fourth embodiment is performed. The selection optimization process is the same as that of the third embodiment.

In the image selection unit 47 according to the fourth embodiment, n actual low-resolution images are input to the image selection unit 47 via the memory 40. The motion amount calculation unit 45 calculates a motion amount between the images F _i and F _j, and the motion amount calculated by the motion amount calculation unit 45 is stored in the motion amount storage unit 46. Here, i and j are integers of 1 or more and n or less. However, i ≠ j. Based on each motion amount stored in the motion amount storage unit 46, the pixel positional relationship of the images F _{1 to} F _n on the common coordinate system is determined.

The image selection unit 47 selects m actual low resolution images from the _n actual low resolution images F _{1 to} F _n by the selection optimization process based on each motion amount stored in the motion amount storage unit 46. Then, the selected m actual low-resolution images are output to the super-resolution processing unit 48 as m calculation target images.

Image selecting section 47 to select the m images (operation target image) from among the n images F ₁ to F _n, evaluate the m images contained in the image F ₁ to F _n of the m sheets An image is extracted, and the suitability for super-resolution processing is evaluated for m evaluation images. The number of combinations that can be taken from n to m is _n C _m . Therefore, the combination of the evaluation image, a combination of the first to _n C _m. In each combination, m evaluation images are formed from the first to mth evaluation images.

The image selection unit 47 performs a polygonal area deriving process for deriving an m-gonal area formed by connecting the target pixel positions of the first to mth evaluation images to each of the combinations of the first to _{nth m m.} The m actual low resolution images corresponding to the largest area among the _n C _m areas derived for the first to _n C _m combinations are executed as m calculation target images. select.

  The polygon area derivation process will be described. First, the target pixel position of the first evaluation image is any one pixel position on the first evaluation image. Similar to the first embodiment, the image selection unit 47 sets a reference rectangular area centered on the target pixel position of the first evaluation image. The pixel positions of the second to mth evaluation images located in the reference rectangular area are the target pixel positions of the second to mth evaluation images. Therefore, if i is considered as an integer of 2 or more and m or less, the target pixel position of the i-th evaluation image is the latest of the target pixel position of the first evaluation image in the pixel position group of the i-th evaluation image. This is a pixel position located nearby.

<< 5th Example >>
A fifth embodiment of the present invention will be described. The video signal processing unit 13 according to the fifth embodiment is the same as that shown in FIG. 5, and the items described in the first embodiment are also applied to the fifth embodiment. In the second to fourth embodiments, when m ≧ 3, the selection optimization process is realized by deriving the polygonal area. However, in the fifth embodiment, the area is not derived, The selection optimization process is realized by deriving the sum of the distances between the vertexes of the squares. Basically, as the area of the polygon increases, the sum of the distances between the vertices of the polygon also increases. Therefore, the fifth embodiment can provide the same effects as those of the second to fourth embodiments.

  In the fifth embodiment, m may be an arbitrary integer equal to or greater than 3, but in the fifth embodiment, it is assumed that m = 3 for the sake of concrete description. Furthermore, it is assumed that n = 4.

In this case, the image selection unit 47 selects three actual low resolutions from among the _four actual low resolution images F _{1 to} F ₄ through the selection optimization process based on each motion amount stored in the motion amount storage unit 46. An image is selected, and the selected three real low-resolution images are output to the super-resolution processor 48 as three calculation target images.

Image selecting section 47 to select the three images (operation target image) from among the four images F ₁ to F _4, evaluation of three, three images included in the image F ₁ to F ₄ Extracted as images, the suitability for super-resolution processing is evaluated for the three evaluation images. The number of combinations that can be taken from n to m is _n C _m . In this example, since n = 4 and m = 3 are assumed, _n C _m = ₄ C ₃ = 4, and a combination of selecting three images from the _four images F _{1 to} F ₄ is used. There are first to fourth combinations.

Similar to the second embodiment, the three evaluation images are represented by symbols F _A , F _B and F _C. Then, in the first, second, third, and fourth combinations, (A, B, C) = (1, 2, 3), (1, 2, 4), (1, 3, 4), respectively. , (2, 3, 4).

The image selection unit 47 performs a sum derivation process for deriving the sum of the distances between the vertices of the triangles connecting the target pixel position of the evaluation image F _A , the target pixel position of the evaluation image F _B , and the target pixel position of the evaluation image F _C. Three real low-resolution images corresponding to the maximum sum among the four sums derived for the first to fourth combinations are executed for each of the first to fourth combinations. Is selected as the calculation target image. Sums derived for the first to fourth combinations are represented by SUM _{1 to} SUM _4, respectively.

The sum derivation process will be described. The significance of the target pixel position of the evaluation images F _{A to} F _C is the same as that described in the second embodiment. When i is 1, 2, 3 or 4, in the sum derivation process for the i-th combination, the distance between the target pixel positions of the evaluation images F _A and F _{B and} the target pixel position of the evaluation images F _B and F _C And the sum SUM _i of the distance between the target pixel positions of the evaluation images F _A and F _C is obtained. Thus, in the example shown in FIGS. 8 and 9, the first combination, the distance between the pixel positions 401a and distance d ₂₃ and the pixel position of the distance d ₁₂ and pixel positions 402a and 403a between 402a 401a and 403a the sum of the _{d 13 (d 12 + d 23} + d 13) is obtained, the sum _{(d 12 + d 23 + d} 13) is the sUM _1. Similarly, for the second combination, the sum (d ₁₂ + d) of the distance d ₁₂ between the pixel positions 401a and 402a, the distance d ₂₄ between the pixel positions 402a and 404a, and the distance d ₁₄ between the pixel positions 401a and 404a. ₂₄ + d ₁₄ ) is obtained, and the sum (d ₁₂ + d ₂₄ + d ₁₄ ) is defined as SUM ₂ . The same applies to the third and fourth combinations.

Then, the image selection unit 47 specifies the maximum sum among the sums SUM _{1 to} SUM ₄ and selects three actual low-resolution images corresponding to the maximum sum as three calculation target images. For example, if the sum SUM ₁ is the largest among the sums SUM ₁ to SUM ₄ , the images F ₁ , F ₂ and F ₃ corresponding to the sum SUM ₁ are selected as three calculation target images, and the sum SUM _{If 4} is the maximum, the images F ₂ , F ₃ and F ₄ corresponding to the sum SUM ₄ are selected as three calculation target images.

Although different from the situation shown in FIG. 8, if the pixel positions 401a, 402a, and 403a are aligned on a straight line, SUM ₁ calculated as the sum (d ₁₂ + d ₂₃ + d ₁₃ ) is the sum of the distances between the vertices of the triangle. I can't call it. The same can be said for the sums SUM _{2 to} SUM ₄ . In such a case, the selection method of the sum SUM _i 3 pieces of arithmetic target image based on the calculation method and the sum SUM _i of are similar to those described above (in the above description, the situation shown in FIG. 8 In consideration of matching, the sum (d ₁₂ + d ₂₃ + d ₁₃ ) is simply called the sum of the distances between the vertices of the triangle).

Further, as described below, the horizontal and vertical distances between the vertices of the polygon may be obtained individually, and the sum of the horizontal and vertical distances may be obtained. Such a process for obtaining the sum is referred to as a modified sum derivation process. When i is 1, 2, 3 or 4, in the deformation sum derivation process for the i-th combination, as shown in FIG. 13, the horizontal distance d _ABH between the _target pixel positions of the evaluation images F _A and F _B , evaluation The horizontal distance d _BCH between the target pixel positions of the images F _B and F _C , the horizontal distance d _ACH between the target pixel positions of the evaluation images F _A and F _C , and the target pixel positions of the evaluation images F _A and F _B. The vertical distance d _ABV , the vertical distance d _BCV between the _target pixel positions of the evaluation images F _B and F _C , and the vertical distance d _ACV between the target pixel positions of the evaluation images F _A and F _C are obtained. Of the distances (d _ABH + d _BCH + d _ACH + d _ABV + d _BCV + d _ACV ) is calculated as the total sum SUM _i . The operation after the sum SUM _i is calculated is the same as described above. In FIG. 13, positions 450A, 450B, and 450C represent the target pixel positions of the evaluation images F _A , F _B, and F _C , respectively.

Note that, unlike the situation shown in FIG. 13, if the positions 450A, 450B, and 450C are arranged in a straight line, SUM _i calculated as the sum (d _ABH + d _BCH + d _ACH + d _ABV + d _BCV + d _ACV ) is a triangle. It cannot be called the horizontal and vertical distance between vertices. However, even in such a case, the method for calculating the sum SUM _i based on the horizontal and vertical distances is the same as that described above.

<< Sixth Example >>
A sixth embodiment of the present invention will be described. In the sixth embodiment, assuming that m is an arbitrary integer equal to or greater than 3, the selection optimization process using the sum derivation process in the fifth embodiment is generalized. Since n> m, n is an integer of 4 or more in the sixth embodiment. If n = 4 and m = 3, the selective optimization process according to the sixth embodiment is the same as that specifically shown in the fifth embodiment.

The image selection unit 47 according to the sixth example receives n actual low resolution images via the memory 40. The motion amount calculation unit 45 calculates a motion amount between the images F _i and F _j, and the motion amount calculated by the motion amount calculation unit 45 is stored in the motion amount storage unit 46. Here, i and j are integers of 1 or more and n or less. However, i ≠ j. Based on each motion amount stored in the motion amount storage unit 46, the pixel positional relationship of the images F _{1 to} F _n on the common coordinate system is determined.

The image selection unit 47 selects m actual low resolution images from the _n actual low resolution images F _{1 to} F _n by the selection optimization process based on each motion amount stored in the motion amount storage unit 46. Then, the selected m actual low-resolution images are output to the super-resolution processing unit 48 as m calculation target images.

Image selecting section 47 to select the m images (operation target image) from among the n images F ₁ to F _n, evaluate the m images contained in the image F ₁ to F _n of the m sheets An image is extracted, and the suitability for super-resolution processing is evaluated for m evaluation images. The number of combinations that can be taken from n to m is _n C _m . Therefore, the combination of the evaluation image, a combination of the first to _n C _m. In each combination, m evaluation images are formed from the first to mth evaluation images.

The image selection unit 47 performs summation derivation processing for deriving the summation of distances between m-vertex vertices formed by connecting the target pixel positions of the first to m-th evaluation images, as a combination of the first to _n-th C _m combinations. The m low resolution images corresponding to the maximum sum among the _n C _m sums derived for the first to _n C _m combinations are executed for each of the first to _n C _m combinations. Select as an image.

The sum derivation process will be described. The significance of the target pixel position of the first to mth evaluation images is the same as that described in the fourth embodiment. When i is an integer greater than or equal to 1 and less than or equal to _n C _m and variables p and q are integers greater than or equal to 1 and less than or equal to m (where p ≠ q), in the summation derivation process for the i-th combination, And the distance between the target pixel positions of the q-th evaluation image are obtained for all combinations of the variables p and q, and the sum of the obtained total distances is the sum to be obtained for the i-th combination. . It is also possible to obtain the sum by the modified sum derivation process described in the fifth embodiment.

  Further, as described in the fifth embodiment, depending on the positional relationship of the target pixel positions of the first to m-th evaluation images (for example, when they are aligned on a straight line), the total of the above total distances is added. Although the value cannot be called the sum of the distances between the apexes of the m-gon, even in such a case, the method for calculating the sum and the method for selecting m calculation target images based on the sum are the same as those described above. .

<< Seventh Embodiment >>
A seventh embodiment of the present invention will be described. In the seventh embodiment, the selection optimization process is performed based not only on the pixel positional relationship of the n actual low resolution images on the common coordinate system but also on the direction of the motion amount. The video signal processing unit 13 according to the seventh embodiment is the same as that shown in FIG. 5, and the items described in the first embodiment are also applied to the seventh embodiment. Although n may be any number as long as it is 4 or more, it is assumed that n = 4 for the sake of concrete description. m may be any number as long as it is greater than or equal to 2 and less than n, but for the sake of concrete explanation, it is assumed that m = 3. In this case, the image selection unit 47 selects three actual low resolutions from among the _four actual low resolution images F _{1 to} F ₄ through the selection optimization process based on each motion amount stored in the motion amount storage unit 46. An image is selected, and the selected three real low-resolution images are output to the super-resolution processor 48 as three calculation target images.

Image selecting section 47 to select the three images (operation target image) from among the four images F ₁ to F _4, evaluation of three, three images included in the image F ₁ to F ₄ Images F _A , F _B and F _C are extracted and the suitability for the super-resolution processing is evaluated for the three evaluation images. There are _first to _fourth combinations for selecting three evaluation images from the _four images F _{1 to} F ₄ . In the first, second, third, and fourth combinations, (A, B, C) = (1, 2, 3), (1, 2, 4), (1, 3, 4), ( 2, 3, 4).

The image selection unit 47 executes an evaluation value calculation process for deriving an evaluation value according to suitability for the super-resolution process for each of the first to fourth combinations, and for the first to fourth combinations. Among the four evaluation values derived in this way, three actual low resolution images corresponding to the maximum evaluation value are selected as three calculation target images. Evaluation values derived for the first to fourth combinations are represented by EV _{1 to} EV _4, respectively.

The evaluation value calculation process will be described. The image selection unit 47 uses, based on the motion amount stored in the motion storage unit 46, the average motion amount between the evaluation images F _A and F _B and the motion amount between the images F _B and F _C as a reference motion amount. calculate. Further, as shown in FIG. 14, the direction connecting the lower left and the upper right of the image is referred to as a diagonal right direction, and the direction connecting the lower right and the upper left of the image is referred to as a diagonal left direction. The vertical direction of the image corresponds to the up and down direction, and the horizontal direction of the image corresponds to the left and right direction.

The direction closest to the calculated reference motion amount direction is selected from the right diagonal direction, left diagonal direction, vertical direction, and horizontal direction.
When the selected direction is the right diagonal direction, the right diagonal direction is set as the first direction, and the left diagonal direction orthogonal thereto is set as the second direction.
When the selected direction is the left diagonal direction, the left diagonal direction is set as the first direction, and the right diagonal direction orthogonal to the selected direction is set as the second direction.
When the selected direction is the up-down direction, the up-down direction is set as the first direction, and the left-right direction orthogonal thereto is set as the second direction.
When the selected direction is the left-right direction, the left-right direction is set as the first direction, and the up-down direction perpendicular thereto is set as the second direction.

After determining the first and second directions, the image selection unit 47 determines the distance along the second direction between the target pixel positions of the evaluation images F _A and F _{B and} the target pixel position of the evaluation images F _A and F _C. The sum total with the distance along the second direction is obtained. The sum obtained for the i-th combination is treated as the evaluation value EV _i . The significance of the target pixel position of the evaluation images F _{A to} F _C is the same as that described in the second embodiment.

  As an example, the evaluation value calculation process for the first combination when the first direction is set to the right diagonal direction will be described. When the dominant direction of image movement, that is, the first direction is the right diagonal direction, since there is a relatively large image movement in the right diagonal direction, the shading as shown in FIG. An edge that changes along the line (hereinafter referred to as a right diagonal edge) is blurred. Therefore, in such a case, even if super-resolution processing is performed so as to increase the resolution (resolution) of the right oblique edge, the effect is small. Accordingly, when the first direction is the right diagonal direction, the resolution of the edge whose shading changes along the left diagonal direction (hereinafter referred to as the left diagonal edge) as shown in FIG. aim.

When the first direction is the right oblique direction, the second direction involved in the derivation of the distance is the left oblique direction as described above. Therefore, the sum of the distance d _AC2 along the left oblique direction between the target pixel position of the distance d _AB2 image F ₁ and F ₃ along the left oblique direction between the target pixel position of the image F ₁ and F ₂ is determined The sum (d _AB2 + d _AC2 ) is treated as the evaluation value EV ₁ . This increase in total (d _AB2 + d _AC2) leads to an increase in the left oblique direction of the amount of information included in the image information of the image F ₁ to F _3, the more the total sum to be increased, the image F ₁ to F ₃ The left diagonal edge on the high-resolution image obtained by using approaches the ideal one.

  Similar evaluation value calculation processing is performed for the second to fourth combinations, and as described above, three actual low-resolution images corresponding to the maximum evaluation value are selected as three calculation target images. . As a result, an optimum actual low-resolution image corresponding to the direction of motion of the image is selected as the calculation target image, and the substantial resolution of the high-resolution image is maximized.

In the above description, it is assumed that m = 3. However, when m = 2, the following may be performed. When m = 2, the image selection unit 47 extracts two images included in the images F _{1 to} F ₄ as two evaluation images F _A and F _B , and for the two evaluation images, Evaluate suitability for super-resolution processing. There are first to sixth combinations for selecting two evaluation images from the _four images F _{1 to} F ₄ . In the first, second, third, fourth, fifth and sixth combinations, (A, B) = (1,2), (1,3), (1,4), (2, 3), (2, 4) and (3,4).

The image selection unit 47 performs an evaluation value calculation process for deriving an evaluation value according to the suitability for the super-resolution process for each of the first to sixth combinations. Evaluation values derived for the first to sixth combinations are represented by EV _{1 to} EV _6, respectively.

In the case of m = 2, the first and second directions are set according to the method described above after handling the motion amount between the evaluation images F _A and F _B as the reference motion amount. Then, the image selection unit 47 obtains a distance along the second direction between the target pixel positions of the evaluation images F _A and F _B. The distance obtained for the i-th combination is treated as the evaluation value EV _i . Then, the image selection unit 47 selects two actual low-resolution images corresponding to the maximum evaluation value among the evaluation values EV _{1 to} EV ₆ as two calculation target images. As a result, it is possible to increase the resolution of the edge so that a change in shading appears along a direction orthogonal to the dominant direction of the image motion.

<< Eighth Example >>
An eighth embodiment of the present invention will be described. In the eighth embodiment, a modified example of the selection optimization process described in each of the above embodiments will be described. Therefore, the matters described in the first to seventh embodiments are appropriately incorporated into the eighth embodiment.

  In the first embodiment, it is assumed that only one distance between target pixel positions is calculated for m evaluation images, but the distance calculation process in the first embodiment is modified as follows. You can also This modified calculation process is called a deformation calculation process.

First, in the eighth embodiment, the motion amount calculated between the two actual low-resolution images by the motion amount calculation unit 45 is the motion vector (motion) calculated for various positions on the image coordinate system. It is assumed that it is formed from a bundle of (quantity). Images 501 and 502 in FIG. 16B represent two actual low-resolution images to be compared for motion amount calculation, and an image 500 in FIG. 16A is an image 501 or 502. Now, as shown in FIG. 16 (a), divided the entire image area is nine part of the image 500 in the image area AR ₁ to Ar _9, 1 single motion with respect to each of the partial image regions AR ₁ to Ar ₉ Assume that a vector is required. However, the number of partial image areas can be other than nine. Then, as shown in FIG. 16 (b), were determined for partial image regions AR ₁ to Ar _9, the motion vector between the images 501 and 502, respectively represented by the symbol M ₁ ~M _9. Each of the motion vectors M _{1 to} M ₉ has a so-called sub-pixel resolution that is higher in resolution than the pixel interval of the actual low-resolution image, like the amount of motion described in the first embodiment. Each obtained motion vector is stored in the motion amount storage unit 46.

In the first embodiment, the entire lattice positions of the images F ₁ and F ₂ are determined based on the motion amount MV _12. However, in the deformation calculation process, this is performed for each partial image region. That is, based on the motion vector M ₁ between the image F ₁ and F ₂ for the partial image area AR _1, so that the position deviation of the image F ₂ on the basis of the image F ₁ is canceled in some image areas AR _1, The grids of the images F ₁ and F ₂ in the partial image area AR ₁ are set on the common coordinate system. The same lattice setting process is also performed for the partial image areas AR _{2 to} AR ₉ . Further, the same lattice setting process is performed for two real low-resolution images other than the images F ₁ and F ₂ . Such a grid setting process for each partial image region is referred to as a modified grid setting process for convenience.

Thereafter, a process for obtaining the distance between the target pixel position of the evaluation image F _A and the target pixel position of the evaluation image F _B is individually performed for each of the partial image areas AR _{1 to} AR ₉ , and the _nine distances thus determined are obtained. Find the total distance. The calculation for obtaining the total distance is executed for each of the first to sixth combinations. Then, the image selection unit 47 sets two actual low-resolution images corresponding to the maximum total distance among the six total distances obtained for the first to sixth combinations as two calculation target images. select.

Note that the processing for obtaining the distance between the target pixel position of the evaluation image F _A and the target pixel position of the evaluation image F _B is performed individually for each of the partial image areas AR _{1 to} AR ₉ , and for each partial image area. It is also possible to select a calculation target image. This process of selecting a calculation target image for each partial image area is referred to as a deformation selection process.

In the deformation selection process, after executing the deformation grid setting process, a process for obtaining the distance between the target pixel position of the evaluation image F _A and the target pixel position of the evaluation image F _B is performed for each of the partial image areas AR _{1 to} AR _9. And for each of the first to sixth combinations. After that, regarding the partial image area AR ₁ , the maximum distance among the six distances obtained for the first to sixth combinations is specified, and two actual low-resolution images corresponding to the maximum distance are identified. selected as two operation target image relative to parts image area AR _1. Similarly, two calculation target images for the partial image areas AR _{2 to} AR ₉ are selected. As a result, two calculation target images for a certain partial image area may be different from two calculation target images for another partial image area.

For example, when the images F ₁ and F ₂ are selected as the calculation target images for the partial image area AR ₁ and the images F ₃ and F ₄ are selected as the calculation target images for the partial image areas AR _{2 to} AR ₉ , The image data in the partial image area AR ₁ of F ₁ and F ₂ and the image data in the partial image areas AR _{2 to} AR ₉ of the images F ₃ and F ₄ are output to the super-resolution processor 48. The super-resolution processing unit 48 generates the entire image data of the high-resolution image by performing the super-resolution processing for each partial image region. In this case, the partial area of the high-resolution image corresponding to the partial image area AR ₁ is generated based on the image data of the portion of the image F ₁ and F ₂ image regions AR _1, partial image areas AR ₂ ~ partial region of the high-resolution image corresponding to AR ₉ will be generated based on the image data of the partial image F ₃ and F ₄ image area AR ₂ to Ar _9.

The deformation calculation process described above can also be applied to other embodiments other than the first embodiment. For example, when the deformation calculation process is applied to the fourth embodiment, the process of deriving the area of the m-gon formed by connecting the target pixel positions of the first to m-th evaluation images after executing the deformation grid setting process. This is performed individually for each of the partial image areas AR _{1 to} AR ₉ , and the total area of the nine areas obtained is obtained. The calculation for obtaining the total area is executed for each of the first to _{n th} C _m combinations. The image selection unit 47 then selects m actual low-resolution images corresponding to the maximum total area among the _n C _m total areas obtained for the first to _{n-th n} C _m combinations. Is selected as the calculation target image.

In addition, when the deformation calculation process is applied to the sixth embodiment, the sum of the m-vertex inter-vertex distances formed by connecting the target pixel positions of the first to m-th evaluation images after the deformation grid setting process is executed. The derivation process is performed individually for each of the partial image areas AR _{1 to} AR ₉ , and the total value of the nine sums obtained is obtained. The calculation for obtaining the total value is executed for each of the first to _{n th} C _m combinations. The image selection unit 47 then selects m actual low-resolution images corresponding to the maximum total value among the _n C _m total values obtained for the first to _{n-th n} C _m combinations. Is selected as the calculation target image.

The deformation selection process described above can be applied to other embodiments other than the first embodiment. For example, when the deformation selection process is applied to the fourth embodiment, the process of deriving the area of the m-gon formed by connecting the target pixel positions of the first to m-th evaluation images after executing the deformation grid setting process. some image areas AR ₁ and with respect to each of the to Ar ₉ people run for the combination of each of the first to _n C _m. Thereafter, for the partial image area AR _i , the maximum area among the _n C _m areas obtained for the first to _{n th n m m} combinations is specified, and m actual lows corresponding to the maximum area are identified. The resolution image is selected as m calculation target images for the partial image area AR _i . This selection is performed for i = 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively. Image data of the calculation target image selected for each partial image area is output to the super-resolution processing unit 48, and a high-resolution image based on the image data is generated by the super-resolution processing.

When the deformation selection process is applied to the sixth embodiment, the sum of the m-vertex inter-vertex distances formed by connecting the target pixel positions of the first to m-th evaluation images after the deformation grid setting process is executed. The derivation process is executed for each of the partial image areas AR _{1 to} AR _{9 and} for each of the first to _{n th} C _m combinations. After that, regarding the partial image area AR _i , the maximum value among the _n C _m totals obtained for the first to _{n-th n} C _m combinations is specified, and m actual lows corresponding to the maximum value are identified. The resolution image is selected as m calculation target images for the partial image area AR _i . This selection is performed for i = 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively. Image data of the calculation target image selected for each partial image area is output to the super-resolution processing unit 48, and a high-resolution image based on the image data is generated by the super-resolution processing.

<< Ninth Embodiment >>
A ninth embodiment of the present invention will be described. In the ninth embodiment, a configuration example of the super-resolution processing unit 48 in FIG. 5 will be described. FIG. 17 is an internal block diagram of the super-resolution processor 48 according to the ninth embodiment. The super-resolution processing unit 48 of FIG. 17 includes a part referred to by reference numerals 61 to 65.

  The initial high-resolution estimation unit 61 is based on the m actual low-resolution images as the m calculation target images output from the image selection unit 47 in FIG. 5 and the motion amount stored in the motion amount storage unit 46. Generate an initial high resolution image. The super-resolution processing unit 48 sets one of m actual low-resolution images as m calculation target images as a reference frame and each of the remaining (m−1) images as a reference frame. .

  The initial high resolution estimator 61 detects a positional shift of each reference frame with respect to the base frame based on the motion amount stored in the motion amount storage unit 46, and performs a positional shift correction for canceling the positional shift. Then, an initial high-resolution image is generated by combining the base frame and the reference frame after the positional deviation correction. As a method for generating an initial high-resolution image, a method using an interpolation process as described in JP-A-2006-41603 can be used.

  The pixel position of the high resolution image generated by the super-resolution processing unit 48 is set based on the pixel position of the reference frame. FIG. 18 shows the positional relationship between the pixels of the reference frame and the pixels of the high resolution image on the common coordinate system. This positional relationship is a positional relationship when the high-resolution image has twice the resolution of the low-resolution image in the horizontal and vertical directions. In FIG. 18, four white circles represent four pixels on the reference frame, and nine white triangles represent nine pixels on the high resolution image. In the example illustrated in FIG. 18, the arrangement positions of the four pixels on the reference frame coincide with the arrangement positions of the four pixels on the high resolution image. That is, the high-resolution image includes the first type pixel and the second type pixel, and the first type pixel is arranged at the same position as the position where the pixel of the reference frame is arranged in the common coordinate system. Then, interpolation is performed at the intermediate position of the first type pixel adjacent in the horizontal direction, the intermediate position of the first type pixel adjacent in the vertical direction, and the intermediate position of the first type pixel adjacent in the oblique direction, respectively. A second type of pixel that is a pixel is arranged.

  The selection unit 62 selects and outputs either the high resolution image (initial high resolution image) generated by the initial high resolution estimation unit 61 or the high resolution image temporarily stored in the frame memory 65. . The selection unit 62 selects the initial high resolution image estimated by the initial high resolution estimation unit 61 in the first selection operation, and the high resolution image temporarily stored in the frame memory 65 in the second and subsequent selection operations. Select.

  The high-resolution update amount calculation unit 63 (hereinafter abbreviated as the update amount calculation unit 63) is stored in the high-resolution image selected by the selection unit 62, m calculation target images, and the motion amount storage unit 46. An update amount for the high-resolution image is obtained based on the amount of movement. More specifically, m camera parameter matrices based on the motion amount stored in the motion amount storage unit 46 are assigned to the matrix X in which the pixel values of the high-resolution image selected by the selection unit 62 are arranged. By multiplying, m estimated low-resolution images corresponding to estimated images of m calculation target images are generated. Then, for example, according to the MAP method, an evaluation function I corresponding to an error between the m estimated low-resolution images and m calculation target images (m actual low-resolution images) is defined. The update amount for the high resolution image is obtained so as to be minimized.

  The subtracting unit 64 updates the high resolution image by subtracting the update amount calculated by the update amount calculating unit 63 from the matrix X of the pixel values of the high resolution image selected by the selecting unit 62. The updated high resolution image has, as a pixel value, an element of a matrix X ′ obtained by subtracting the update amount from the matrix X. The updated high resolution image is given to the frame memory 65. The frame memory 65 temporarily stores the high-resolution image after the image update according to the update amount given from the subtraction unit 64, and gives this to the selection unit 62. Thereby, the high-resolution image output from the subtraction unit 64 is updated again by the update amount calculation unit 63 and the subtraction unit 64.

  The super-resolution calculation process including the high-resolution image update process by the update amount calculation unit 63 and the like is repeatedly executed. An upper limit number can be set for the number of repetitions. When the number of repetitions of the super-resolution calculation process reaches the upper limit, the latest high-resolution image obtained by the subtracting unit 64 at that time is output from the subtracting unit 64 to the signal processing unit 49 in FIG. If it is determined that the update amount for the high-resolution image has become sufficiently small regardless of the number of times the super-resolution calculation process is repeated, the latest high-resolution image obtained by the subtracting unit 64 at that time. May be output from the subtracting unit 64 to the signal processing unit 49.

  The configuration example of the super-resolution processing unit that repeatedly executes the super-resolution calculation processing has been described. As described above, the initial high-resolution image should be finally obtained without performing this iteration. Can be output to the signal processing unit 49 as well.

<< Tenth Embodiment >>
A tenth embodiment of the present invention will be described. In the above description, image processing including super-resolution processing is performed in the imaging apparatus 1 which is a kind of electronic apparatus. However, the image processing can be realized by an electronic apparatus different from the imaging apparatus 1. It is. This electronic apparatus is, for example, an image reproduction apparatus 101 that can reproduce an image as shown in FIG. The image reproduction apparatus 101 includes a video signal processing unit (image processing unit) 102 having the same configuration and functions as the video signal processing unit 13 in FIG. 5 and a display unit 103 including a liquid crystal display panel. Image data of an actual low-resolution image sequence is supplied from the outside of the image reproduction apparatus 101 to the video signal processing unit 102. For example, image data of n actual low-resolution images acquired by the imaging device 1 or another imaging device is supplied to the video signal processing unit 102 wirelessly or via a wire or via a recording medium. The video signal processing unit 102 executes selection optimization processing and super-resolution processing on the n actual low-resolution images, thereby generating a high-resolution image. This high resolution image can be reproduced and displayed on the display unit 103.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The example in which the high-resolution image has twice the resolution of the low-resolution image, that is, the example in which the resolution enlargement ratio of the high-resolution image with respect to the low-resolution image is doubled is described above. It may be other than.

[Note 2]
In the above description, the amount of motion between real low-resolution images is derived by computation based on image data, but based on the detection result of a sensor (not shown) that detects the motion of the imaging device 1 in real space, You may make it derive | lead-out the motion amount between real low-resolution images. The sensor that detects the movement of the imaging device 1 is, for example, an angular velocity sensor that detects an angular velocity of the imaging device 1, an angular acceleration sensor that detects angular acceleration of the imaging device 1, or an acceleration sensor that detects acceleration of the imaging device 1, Or a combination thereof. Further, the amount of motion between actual low-resolution images may be derived based on both the detection result of such a sensor and the image data.

[Note 3]
Each of the imaging device 1 in FIG. 1 and the image reproduction device 101 in FIG. 19 can be realized by hardware or a combination of hardware and software. In particular, a part of the processing executed in the video signal processing unit 13 or 102 can be realized using software. Of course, the video signal processing unit 13 or 102 can be formed only by hardware. When the imaging apparatus 1 or the image playback apparatus 101 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram of the reconstruction type super-resolution process based on the MAP method using an iterative operation. 3 is a flowchart showing a flow of super-resolution processing corresponding to FIG. 2. It is a figure for demonstrating the basic selection guideline of the selection optimization process which concerns on embodiment of this invention, Comprising: The figure on the 1st-3rd real low-resolution image arrange | positioned and showed in the common coordinate system. is there. FIG. 2 is a block diagram of a portion related to super-resolution processing in the imaging apparatus of FIG. 1 according to the embodiment of the present invention. It is the figure which showed the amount of motion calculated | required with respect to four real low-resolution images concerning embodiment of this invention. FIG. 6 is a diagram illustrating a state where four pixels of an actual low resolution image are arranged on a common coordinate system based on a detected motion amount according to the first embodiment of the present invention. FIG. 6 is a diagram for explaining a distance between pixel positions according to the first embodiment of the present invention. FIG. 6 is a diagram for explaining a distance between pixel positions according to the first embodiment of the present invention. It is the figure which showed a mode that the area of the triangle which has 3 vertices in three attention pixel positions was maximized concerning 2nd Example of this invention. It is a figure which concerns on 2nd Example of this invention and shows the example of the pixel positional relationship on a common coordinate system. FIG. 10 is a diagram illustrating a state in which the area of a quadrangle having four vertices at four target pixel positions is maximized according to the third embodiment of the present invention. It is a figure for demonstrating the meaning of the horizontal direction distance and the vertical direction distance which concerns on 5th Example of this invention. It is a figure which concerns on 7th Example of this invention and shows the example of the pixel positional relationship on a common coordinate system. It is a figure showing the right diagonal edge (a) and left diagonal edge (b) which may exist on an image. FIG. 14A is a diagram illustrating a state in which an entire image area of one real low-resolution image is divided into nine partial image areas according to the eighth embodiment of the present invention, and two real low-resolution images. FIG. 8B is a diagram illustrating a state in which a motion vector is calculated for each partial image region. It is an internal block diagram of the super-resolution processing part which concerns on 9th Example of this invention. FIG. 30 is a diagram illustrating a positional relationship between a pixel of a reference frame and a pixel of a high resolution image according to the ninth embodiment of the present invention. It is an internal block diagram of the image reproduction apparatus which concerns on 10th Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Image | video signal processing part 33 Image pick-up element 45 Motion amount calculation part 46 Motion amount memory | storage part 47 Image selection part 48 Super-resolution processing part

Claims (6)

an image input unit for receiving n low-resolution images (n is an integer of 3 or more);
A motion amount detection unit for detecting a motion amount between different low resolution images included in the n low resolution images;
Based on the detection result of the motion amount detection unit, an image selection unit that selects m of the n low resolution images as m calculation target images (m is an integer of 2 or more, and n> m is established)
An image processing apparatus comprising: a high-resolution processing unit that generates a high-resolution image having a resolution higher than the resolution of the low-resolution image based on the m calculation target images and the detection result of the motion amount detection unit. Because
The image selection unit performs an evaluation based on a detection result of the motion amount detection unit for each of _n C _m combinations obtained by taking m images from the n low-resolution images. An image processing apparatus, wherein the m calculation target images are selected by selecting one combination. When m low-resolution images for each combination are called m evaluation images,
The image selection unit is configured to combine pixel positional relationships between the m evaluation images when the evaluation images are arranged in a common coordinate system based on the detected motion amount between the m evaluation images. The image processing apparatus according to claim 1, wherein the m calculation target images are selected by evaluating the image and selecting one combination from the evaluation result. The m evaluation images include first and second evaluation images,
In each combination, the target pixel position on the second evaluation image is a pixel arranged in the nearest vicinity of the target pixel position on the first evaluation image in the pixel position group on the second evaluation image. Position,
The image selection unit performs distance derivation processing for deriving a distance between a target pixel position on the first evaluation image and a target pixel position on the second evaluation image for each combination, Among the distances derived for the combination, the distance closest to the predetermined distance based on the distance between adjacent pixels of the low-resolution image is determined, and the m evaluation images corresponding to the closest distance are determined. The image processing device according to claim 2, wherein the image processing device is selected as a calculation target image. m is 3 or more, and the m evaluation images include first to m-th evaluation images,
In each combination, the pixel position of interest on the i-th evaluation image is a pixel position arranged in the nearest vicinity of the pixel position of interest on the first evaluation image in the pixel position group on the i-th evaluation image. (I is an integer from 2 to m),
The image selection unit executes an area derivation process for deriving an area of a polygon formed by connecting m target pixel positions for the m evaluation images with respect to each combination. The image processing apparatus according to claim 2, wherein the m calculation target images are selected by selecting one combination based on the size relationship of the areas derived in this way. m is 3 or more, and the m evaluation images include first to m-th evaluation images,
In each combination, the pixel position of interest on the i-th evaluation image is a pixel position arranged in the nearest vicinity of the pixel position of interest on the first evaluation image in the pixel position group on the i-th evaluation image. (I is an integer from 2 to m),
The image selection unit pays attention to a plurality of sets of two target pixel positions included in the m target pixel positions for the m evaluation images, and a distance between the two target pixel positions for each set. And a sum derivation process for deriving the sum of the obtained distances is performed for each combination, and one combination is selected based on the magnitude relation of the sum derived for each combination. The image processing apparatus according to claim 2, wherein one calculation target image is selected. An image processing device,
An electronic device that obtains n images by photographing or input from the outside, and supplies an image signal of the n images to the image processing device,
Using the image processing device according to any one of claims 1 to 5 as the image processing device,
The image processing apparatus receives the n images as the n low-resolution images.

Images