WO2010032649A1

WO2010032649A1 - Image display device and imaging apparatus

Info

Publication number: WO2010032649A1
Application number: PCT/JP2009/065609
Authority: WO
Inventors: 法和恒川; 誠司岡田
Original assignee: 三洋電機株式会社
Priority date: 2008-09-16
Filing date: 2009-09-07
Publication date: 2010-03-25
Also published as: JP2010072092A

Abstract

A super-resolution operation unit generates a high-resolution image from multiple low-resolution images by reconstruction-based super-resolution processing using a repeated operation. The super-resolution processing comprises super-resolution unit processing which is repeatedly executed. The super-resolution operation unit sequentially updates the high-resolution image estimated from the multiple low-resolution images by repeat execution the super-resolution unit processing and outputs the high-resolution image obtained after a predetermined number of updates as a final high-resolution image. The execution of the super-resolution processing is started while one low-resolution image (301) is being displayed immediately after the multiple low-resolution images are captured. Thereafter, while the super-resolution unit processing is repeatedly executed, an intermediate high-resolution image (303, 304) obtained at that point in time is displayed, and after the completion of the super-resolution processing, the final super-resolution image (302) is displayed.

Description

Image display device and imaging device

The present invention relates to an image display device and an imaging device capable of displaying an image.

High resolution processing (super-resolution processing) that generates one high-resolution image from a plurality of low-resolution images has been proposed. Among the super-resolution processes, a reconfiguration-type super-resolution process using repetitive calculations (repetitive calculations) is known as a typical process. The reconfiguration-type super-resolution processing method using iterative computation is the most effective method among the super-resolution processing methods at present, but it is an optimization computation method that requires many iterative computations. It takes a relatively long time.

On the other hand, the user wishes to confirm the captured image promptly on the display screen. Therefore, for example, in order to obtain a high-definition still image, even when a plurality of frame images (low-resolution images) for generating a high-resolution image are captured, an image based on the captured image should be displayed promptly. .

In order to achieve this, in a conventional method, after capturing a plurality of frame images, one of the plurality of frame images is immediately displayed on the display unit, while the super solution using the plurality of frame images is displayed. After the image processing is executed and the super-resolution processing is completed, the obtained high-resolution image is stored in a recording medium (for example, see Patent Document 1 below).

JP 2002-112103 A

However, in this conventional method, as shown in FIG. 24, there is a large difference in image quality between a display image (low resolution image) that is confirmed by the user immediately after shooting and a saved image (high resolution image) that is saved afterwards. Therefore, the user often feels uncomfortable when viewing the stored image later (in FIG. 24, the effect of the super-resolution processing is exaggerated). Also, with this conventional method, the user cannot confirm the processing result at all until the super-resolution processing that requires a relatively long time is completed. This is contrary to the user's desire to confirm the effect of high resolution based on super-resolution processing as soon as possible. If the result of the super-resolution processing can be presented as early as possible for the user, these problems are solved or reduced. Note that the conventional problem has been described by focusing on the case where the image processing to be executed is the super-resolution process, but the same problem exists when image processing other than the super-resolution process is executed.

Therefore, an object of the present invention is to provide an image display device and an imaging device that can present the result of image processing as early as possible to the user.

An image display device according to the present invention includes: an arithmetic processing unit that generates an output image from an input image by predetermined arithmetic processing; and a display control unit that displays a display image based on the generated image of the arithmetic processing unit on the display unit. In the image display apparatus provided, the display control unit causes the display unit to display an intermediate result of the calculation process generated in the execution process of the calculation process in a stepwise manner during the execution of the calculation process. Features.

This makes it possible to present the results of image processing (arithmetic processing) as early as possible for the user.

Specifically, for example, the arithmetic processing includes unit processing that is repeatedly executed, and the arithmetic processing unit repeatedly executes the unit processing on the intermediate generated image based on the input image to thereby generate the intermediate generated image. And finally generating the output image, and when the unit process is repeatedly executed, the display control unit generates a display image from the intermediate generated image in the iteration process of the unit process, and Display on the display.

Further, for example, when the unit process is repeatedly executed, the display control unit gradually changes the display unit according to the elapsed time from the execution start time of the arithmetic process or the number of times the unit process is repeatedly executed. And the latest intermediate generation image at the time of update is reflected in the display content.

Alternatively, for example, the arithmetic processing unit repeatedly executes the unit processing in order to improve the image quality of the intermediate generated image and sequentially updates the intermediate generated image, and the image display device is based on the repeated execution of the unit processing. The display unit further includes an estimation unit that estimates an image quality improvement amount of the intermediate generation image, and the display control unit displays the display content of the display unit step by step according to the estimated image quality improvement amount when the unit processing is repeatedly executed. And the latest intermediate generation image at the time of update is reflected in the display content.

Further, for example, the calculation process includes first to n-th unit processes, and an i-th intermediate generation image based on a part of the input image is generated as a part of the output image by the i-th unit process. The entire output image is formed by synthesizing the first to n-th intermediate generated images generated by the first to n-th unit processes, and the display control unit performs the calculation process , A display image is generated using the first to m-th intermediate generation images obtained at that time and displayed on the display unit, where n and m are natural numbers, and n> m holds, and i is It is an integer that satisfies 1 ≦ i ≦ n.

An imaging apparatus according to the present invention includes an imaging unit that acquires an image by imaging and the image display apparatus. The image display device receives the image acquired by the imaging unit as the input image.

According to the present invention, it is possible to provide an image display device and an imaging device capable of presenting the result of image processing as early as possible to the user.

The significance or effect of the present invention will be further clarified by the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram of the reconstruction type super-resolution process based on the MAP method using an iterative operation. 3 is a flowchart showing a flow of super-resolution processing corresponding to FIG. 2. It is a figure which shows the example of the display image of the periphery at the time of execution of a super-resolution process. It is a figure which shows the analog image of the to-be-photographed object which should be image | photographed with the imaging device of FIG. It is a figure which concerns on embodiment of this invention and shows the example of the display image before and behind execution of a super-resolution process, and execution. FIG. 3 is an internal block diagram of a video signal processing unit according to the first embodiment of the present invention. FIG. 8 is an internal block diagram of a super-resolution operation unit in FIG. 7. It is a figure which shows a mode that the period of the periphery at the time of execution of a super-resolution process is classified into three periods concerning 1st Example of this invention. FIG. 6 is a diagram illustrating a relationship between an elapsed time from the start of execution of super-resolution processing and display update processing execution timing according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating a relationship between the number of executions of super-resolution unit processing and the execution timing of display update processing according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating a relationship between the number of executions of super-resolution unit processing and an improvement evaluation value representing an image quality improvement amount by repeated execution of super-resolution unit processing according to the first embodiment of the present invention. It is an internal block diagram of the display processing part which concerns on 1st Example of this invention. (A) And (b) is a figure which shows the figure showing the whole high resolution image, and a display image, respectively, concerning 1st Example of this invention. It is a figure which shows a mode that an additional item is superimposed on a display image concerning 1st Example of this invention. It is a figure which concerns on 1st Example of this invention and shows a mode that another additional item is superimposed on a display image. It is a figure which concerns on 1st Example of this invention and shows a mode that another additional item is further superimposed on a display image. (A) And (b) concerns on 1st Example of this invention, and is a figure which shows a mode that another additional item is further superimposed on a display image, respectively, and a figure which shows the icon showing the additional item. . (A) And (b) is a figure which concerns on 1st Example of this invention and shows the example of the pixel positional relationship between the two low-resolution images after alignment. It is an internal block diagram of the super-resolution processing part which concerns on 2nd Example of this invention. It is a figure which shows a mode that the whole image is divided | segmented into a several area | region concerning 2nd Example of this invention. It is a figure which shows the flow of the super-resolution process based on 2nd Example of this invention. It is a figure which concerns on 2nd Example of this invention and shows the mode of a transition of a display image. It is a figure which shows the display image (low-resolution image) which a user confirms immediately after imaging | photography and the preservation | save image (high-resolution image) preserve | saved after the fact regarding a prior art.

Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to third embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera, for example. The imaging device 1 can capture a moving image and a still image, and can also capture a still image simultaneously during moving image capturing.

[Description of basic configuration]
The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and a DRAM (Dynamic Random Access Memory). An internal memory 17 such as an SD (Secure Digital) card or a magnetic disk, an expansion processing unit 19, a display processing unit 20, an audio output circuit 21, a TG (timing generator) 22, and a CPU. (Central Processing Unit) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the

bus

24 or 25.

The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the internal memory 17 during signal processing as necessary.

The imaging unit 11 includes an imaging system (image sensor) 33, an optical system, a diaphragm, and a driver (not shown). Incident light from the subject enters the image sensor 33 via the optical system and the stop. Each lens constituting the optical system forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

The image sensor 33 is a solid-state image sensor composed of a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 33 photoelectrically converts an optical image incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels (not shown in FIG. 1) that are two-dimensionally arranged in a matrix, and in each photographing, each light receiving pixel has a charge amount signal corresponding to the exposure time. Stores charge. The electrical signal from each light receiving pixel having a magnitude proportional to the amount of the stored signal charge is sequentially output to the subsequent AFE 12 in accordance with the drive pulse from the TG 22.

The AFE 12 amplifies an analog signal output from the image sensor 33 (each light receiving pixel), converts the amplified analog signal into a digital signal, and outputs the digital signal to the video signal processing unit 13. The degree of amplification of signal amplification in the AFE 12 is controlled by the CPU 23. The video signal processing unit 13 performs various types of image processing on the image represented by the output signal of the AFE 12, and generates a video signal for the image after the image processing. The video signal is generally composed of a luminance signal Y representing the luminance of the image and color difference signals U and V representing the color of the image.

The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. The compressed video signal is recorded in the external memory 18 at the time of capturing and recording a moving image or a still image. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting and recording, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being correlated with each other in time by the compression processing unit 16, and after compression, Recorded in the external memory 18.

The recording button 26a is a push button switch for instructing start / end of moving image shooting and recording, and the shutter button 26b is a push button switch for instructing shooting and recording of a still image.

The operation mode of the imaging apparatus 1 includes a shooting mode capable of shooting a moving image and a still image, and a playback mode for reproducing and displaying the moving image and the still image stored in the external memory 18 on the display unit 27. It is. Transition between the modes is performed according to the operation on the operation key 26c.

In the shooting mode, shooting is sequentially performed at a predetermined frame period, and a shot image sequence is acquired from the image sensor 33. An image sequence typified by a captured image sequence refers to a collection of images arranged in time series. Data representing an image is called image data. Image data can also be considered as a kind of video signal. One image is represented by image data for one frame period. One image represented by image data for one frame period is also called a frame image.

When the user presses the recording button 26a in the shooting mode, under the control of the CPU 23, the video signal obtained after the pressing and the corresponding audio signal are sequentially recorded in the external memory 18 via the compression processing unit 16. . When the user presses the recording button 26a again after starting the moving image shooting, the recording of the video signal and the audio signal to the external memory 18 is completed, and the shooting of one moving image is completed. In the shooting mode, when the user presses the shutter button 26b, a still image is shot and recorded.

In the playback mode, when the user performs a predetermined operation on the operation key 26c, a compressed video signal representing a moving image or a still image recorded in the external memory 18 is expanded by the expansion processing unit 19 and displayed. Sent to. In the shooting mode, the generation of the video signal by the video signal processing 13 is normally performed regardless of the operation contents on the recording button 26 a and the shutter button 26 b, and the video signal is sent to the display processing unit 20. .

The display processing unit 20 causes the display unit 27 to display an image corresponding to the given video signal. The display unit 27 is a display device such as a liquid crystal display. In addition, when a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded in the external memory 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

The video signal processing unit 13 is configured to be able to perform super-resolution processing in cooperation with the CPU 23. Through the super-resolution processing, one high-resolution image is generated from a plurality of low-resolution images. The video signal of the high resolution image can be recorded in the external memory 18 via the compression processing unit 16. The resolution of the high resolution image is higher than that of the low resolution image, and the number of pixels in the horizontal and vertical directions of the high resolution image is larger than that of the low resolution image. For example, when an instruction to capture a still image is given, a plurality of frame images as a plurality of low resolution images are acquired, and a high resolution image is generated by performing super-resolution processing on them. Alternatively, for example, the super-resolution processing is performed on a plurality of frame images as a plurality of low resolution images obtained at the time of moving image shooting.

[Basic concept of super-resolution processing]
The basic concept of super-resolution processing will be briefly described. There is a method called a reconstruction type as a type of super-resolution processing. On the other hand, there is a method for realizing super-resolution processing by repetitive calculation (repetitive calculation algorithm), and this repetitive calculation can also be applied to reconstruction-type super-resolution processing. In the present embodiment, a main example is a reconstruction type super-resolution process based on a MAP (Maximum A Posterior) method using a repetitive calculation, which is a kind of super-resolution process that can be realized by a repetitive calculation.

FIG. 2 shows a conceptual diagram of a reconfigurable super-resolution process based on the MAP method using repetitive operations. In this super-resolution processing, one high-resolution image is estimated from a plurality of low-resolution images obtained by actual shooting, and the original plurality of low-resolution images are estimated by degrading the estimated high-resolution image. To do. A low resolution image obtained by actual photographing is particularly called an “real low resolution image”, and an estimated low resolution image is particularly called an “estimated low resolution image”. Thereafter, the high resolution image and the low resolution image are repeatedly estimated so that the error between the actual low resolution image and the estimated low resolution image is minimized, and the finally acquired high resolution image is output.

FIG. 3 is a flowchart showing the flow of super-resolution processing corresponding to FIG. First, in step S11, an initial high resolution image is generated from an actual low resolution image. In subsequent step S12, the original actual low resolution image for constructing the current high resolution image is estimated. The estimated image is referred to as an estimated low resolution image as described above. In the subsequent step S13, an update amount for the current high resolution image is derived based on the difference (difference image) between the actual low resolution image and the estimated low resolution image. This update amount is derived so that the error between the actual low-resolution image and the estimated low-resolution image is minimized by repeatedly executing the processes in steps S12 to S14. In the subsequent step S14, the current high resolution image is updated using the updated amount, and a new high resolution image is generated. Thereafter, the process returns to step S12, and the newly generated high resolution image is regarded as the current high resolution image, and the processes of steps S12 to S14 are repeatedly executed. Basically, as the number of repetitions of each of the processes in steps S12 to S14 increases, the resolution of the obtained high-resolution image is substantially improved (resolution is improved), and an ideal high-resolution image can be obtained.

Super-resolution processing based on the above-described operation flow is performed in the imaging apparatus 1. In addition to super-resolution processing based on the MAP method, super-resolution processing based on the ML (Maximum-Likelihood) method, POCS (Projection Onto Convex Set) method, or IBP (Iterative Back Projection) method can also be used. is there.

The reconstruction-type super-resolution processing method using iterative computation is an effective method, but it takes a relatively long time for processing because it is an optimization computation method that requires many iterative computations. On the other hand, the user desires to confirm the captured image on the display unit 27 promptly. Therefore, for example, in order to obtain a high-definition still image, even when a plurality of frame images (actual low-resolution images) for generating a high-resolution image are captured, an image based on the captured image should be displayed promptly. is there. In order to realize this, after capturing a plurality of frame images, one of the plurality of frame images is immediately displayed on the display unit 27, while super-resolution processing using the plurality of frame images is executed. A method of executing display of the obtained high resolution image on the display unit 27 and storage in the external memory 18 after completion of the super-resolution processing is conceivable.

Fig. 4 shows a conceptual diagram of this method. FIG. 5 is an analog image of a subject to be photographed by the imaging apparatus 1. In FIG. 4, an image 301 is a frame image before super-resolution processing that is displayed immediately after shooting, and an image 302 is a high-resolution image based on a plurality of frame images that is displayed after completion of the super-resolution processing. It is. In FIG. 4, the effect of the super-resolution processing is exaggerated (the same applies to FIG. 6 described later).

In the method corresponding to FIG. 4, the user cannot confirm the processing result at all until the super-resolution processing that requires a relatively long time is completed. This is contrary to the user's desire to confirm the effect of high resolution based on super-resolution processing as soon as possible. In view of the fact that the image 302 can be obtained only after a lapse of a suitable time, the user often confirms only the image 301 on the display unit 27. Even in such a case, after the super-resolution processing is completed, the high-resolution image is stored in the external memory 18, but when the user who has confirmed only the image 301 looks at the stored image later, Since the image 301 and the stored image are considerably different in image quality, there may be a sense of incongruity.

Considering these circumstances, the imaging device 1 displays the intermediate result of the super-resolution processing generated in the process of executing the super-resolution processing on the display unit 27 step by step during the execution of the super-resolution processing. For example, as shown in FIG. 6, the execution of the super-resolution processing is started while displaying the image 301 immediately after capturing a plurality of frame images for generating a high-resolution image, and the processing consists of steps S12 to S14 in FIG. An intermediate high-resolution image that is sequentially updated in the process of repeatedly executing the group is updated and displayed on the display unit 27 step by step.

According to the method corresponding to FIG. 6, after a relatively short time has elapsed since the image 301 was displayed, the

images

303 and 304 that should be called intermediate results of the super-resolution processing are sequentially updated and displayed. After the processing is completed, an image 302 that is a final high-resolution image is displayed. Thereby, although the user is a part of the result of the super-resolution processing, the user can check the result of the super-resolution processing with a short waiting time. Moreover, the possibility of having the above-mentioned uncomfortable feeling is reduced.

Hereinafter, the display method as described above, the contents of the super-resolution processing, and / or technical matters related thereto will be described in the first to third embodiments. As long as no contradiction arises, the matters described in one embodiment can be applied to other embodiments.

In the following description, a frame image obtained by shooting is handled as an actual low resolution image unless otherwise specified. Further, it is assumed that an actual low resolution image Fa is obtained by shooting at a certain time, and an actual low resolution image Fb is obtained by subsequent shooting. The shooting interval between the images Fa and Fb corresponds to, for example, a frame period.

In addition, in this specification, for simplification of description, names are sometimes abbreviated or omitted by using symbols. For example, in this specification, the actual low-resolution image Fa may be simply expressed as “image Fa” or “Fa”, but the former and the latter indicate the same thing.

<< First Example >>
A first embodiment of the present invention will be described. FIG. 7 is an internal block diagram of the video signal processing unit 13 according to the first embodiment. The video signal processing unit 13 in FIG. 7 includes a super-resolution processing unit 40, an iterative control unit 50, first and second

signal control units

51 and 53, and a signal processing unit 52. The super-resolution processing unit 40 includes a memory unit 41 having

frame memories

41A and 41B, a motion amount calculation unit 42, a motion amount storage unit 43, and a super-resolution calculation unit 44.

The frame memory 41A temporarily stores image data of an actual low resolution image for one frame represented by a digital signal from the AFE 12. The frame memory 41B temporarily stores image data of an actual low resolution image for one frame stored in the frame memory 41A. The contents stored in the frame memory 41A are transferred to the frame memory 42B every time one frame elapses. Thereby, at the end of the second frame, the image data of the actual low resolution images Fa and Fb are recorded in the

frame memories

41B and 41A, respectively.

The motion amount calculation unit 42 is provided with the image data of the actual low resolution image of the current frame from the AFE 12 and the image data of the actual low resolution image of the previous frame from the frame memory 41A. The motion amount calculation unit 42 calculates a motion amount representing the amount of positional deviation between two given real low-resolution images by comparing the two given image data. This motion amount is a two-dimensional amount including a horizontal component and a vertical component, and is expressed as a so-called motion vector. The calculated motion amount is stored in the motion amount storage unit 43.

The motion amount calculation unit 42 calculates a motion amount between two real low-resolution images using a representative point matching method, a block matching method, a gradient method, or the like. The motion amount calculated here has a so-called sub-pixel resolution having a resolution higher than the pixel interval of the actual low-resolution image. That is, the amount of motion is calculated with a distance shorter than the interval pp _L between two pixels adjacent in the horizontal or vertical direction in the actual low resolution image as the minimum unit. A known calculation method can be used as a method for calculating a positional deviation amount having sub-pixel resolution. For example, the method described in Japanese Patent Application Laid-Open No. 11-345315 and the method described in “Okutomi,“ Digital Image Processing ”, Second Edition, CG-ARTS Association, issued on March 1, 2007” (p. 40). 205).

The super-resolution calculation unit 44 is based on the two actual low-resolution images Fa and Fb given from the

frame memories

41A and 41B and the motion amount between the images Fa and Fb stored in the motion amount storage unit 43. A high resolution image is generated by super-resolution processing. The super-resolution calculation unit 44 first generates an initial high-resolution image as a high-resolution image before update from the images Fa and Fb according to the processing in steps S11 to S14 in FIG. 3, and then sequentially updates the high-resolution image. To go. The initial high resolution image is represented by symbol Fx1, and the high resolution image obtained by executing the process of step S14 once for the initial high resolution image Fx1 is represented by symbol Fx2. That is, the high-resolution image Fx2 is obtained by updating the initial high-resolution image Fx1 only once. Thereafter, the high resolution images Fx3, Fx4,... Are sequentially obtained by sequentially updating the high resolution image Fx2.

The first signal processing unit 51 outputs the image data of the high resolution image output from the super-resolution calculation unit 44 to the super-resolution calculation unit 44 and / or the signal processing unit 52 under the control of the iterative control unit 50. To do. The signal processing unit 52 generates a video signal (luminance signal and color difference signal) of the high resolution image from the image data of the high resolution image given through the first signal processing unit 51. The second signal processing unit 52 outputs the video signal of the high-resolution image generated by the signal processing unit 52 to the display processing unit 20 and / or the compression processing unit 16 in FIG. 1 under the control of the iterative control unit 50. To do. The iterative control unit 50 controls the first and second

signal processing units

51 and 53. Prior to detailed description of the control, a configuration example of the super-resolution calculation unit 44 will be described.

[Configuration of Super-Resolution Operation Unit 44]
FIG. 8 is an internal block diagram of the super-resolution operation unit 44. The super-resolution operation unit 44 in FIG. 8 includes parts referred to by reference numerals 61 to 65.

In the super-resolution calculation unit 44, one of the two images Fa and Fb is set as a reference frame and the other is set as a reference frame. Assume that the image Fa is set as a reference frame. Further, it is assumed that the number of pixels of the low resolution image is u and the number of pixels of the high resolution image is v. v is an arbitrary value larger than u. For example, if the resolution of the high resolution image is twice that of the low resolution image in the vertical and horizontal directions, v is 4 times u. Of course, the resolution of the high resolution image may be other than twice the resolution of the low resolution image. A matrix in which the pixel values of the actual low resolution image Fa composed of u pixels are written and arranged is represented by Ya, and a matrix in which the pixel values of the actual low resolution image Fb composed of u pixels are arranged and represented is represented by Yb.

The initial high resolution estimation unit 61 executes a process corresponding to step S11 in FIG. The reference frame can be regarded as an image obtained by shifting the position of the reference frame by an amount corresponding to the amount of motion between the reference frame and the reference frame. Therefore, the initial high-resolution estimation unit 61 detects the displacement of the reference frame with respect to the reference frame based on the amount of movement between the reference frame and the reference frame, which is stored in the movement amount storage unit 43, and detects these displacements. The position deviation correction for canceling out is performed. Then, an initial high-resolution image is generated by combining the base frame and the reference frame after the positional deviation correction. As a method for generating an initial high-resolution image, a method using an interpolation process as described in JP-A-2006-41603 can be used.

X represents a matrix in which pixel values of the initial high-resolution image Fx1 composed of v pixels are written and arranged. A matrix in which pixel values of high-resolution images (for example, Fx2) other than the initial high-resolution image Fx1 are arranged is also represented by X. However, the contents of the matrix X of the high resolution image Fxi are different from the contents of the matrix X of the high resolution image Fxj (where i ≠ j).

The selection unit 62 selects and outputs either the high resolution image (initial high resolution image) generated by the initial high resolution estimation unit 61 or the high resolution image temporarily stored in the frame memory 65. . The selection unit 62 selects the initial high resolution image estimated by the initial high resolution estimation unit 61 in the first selection operation, and the high resolution image temporarily stored in the frame memory 65 in the second and subsequent selection operations. Select.

The high-resolution update amount calculation unit (hereinafter abbreviated as update amount calculation unit) 63 is stored in the high-resolution image selected by the selection unit 62, the actual low-resolution images Fa and Fb, and the motion amount storage unit 43. Based on the amount of motion between the images Fa and Fb, the displacement of the actual low-resolution images Fa and Fb with respect to the high-resolution image given from the selection unit 62 is calculated. After that, in order to estimate the original low resolution image (that is, the actual low resolution images Fa and Fb) by degrading the high resolution image from the selection unit 62, each calculated displacement, image blur caused by the low resolution, In addition, camera parameter matrices Wa and Wb using the amount of downsampling from a high resolution image of v pixels to a low resolution image of u pixels as parameters are obtained.

Then, as shown in step S12 of FIG. 3, the update amount calculation unit 63 multiplies the camera parameter matrices Wa and Wb individually by the matrix X of the high resolution image selected by the selection unit 62, thereby Two estimated low resolution images corresponding to the estimated images of the low resolution images Fa and Fb are generated. The two estimated low-resolution images are represented by matrices Wa · X and Wb · X.

The error between the estimated low resolution image and the actual low resolution image is represented by | Wa · X−Ya | and | Wb · X−Yb |. Therefore, an evaluation function I of the following formula (1) is defined as an evaluation function for estimating the error, and an update amount for the high resolution image is obtained so as to minimize the evaluation function I. The third term on the right side of Equation (1) is a constraint term based on the high-resolution image from the selection unit 62. The matrix C in the constraint term γ | C · X | ² is a matrix based on the prior probability model. The matrix C is set based on prior knowledge that “a high-resolution image has few high-frequency components”, and is formed by a high-pass filter such as a Laplacian filter. The coefficient γ is a parameter representing the weight of the constraint term for the evaluation function I.
I = | Wa · X−Ya | ² + | Wb · X−Yb | ² + γ | C · X | ²
... (1)

Although any method can be adopted as a method for minimizing the evaluation function I, it is assumed that the gradient method is used now. In this case, the update amount calculation unit 63 obtains the gradient ∂I / ∂X with respect to the evaluation function I. The gradient ∂I / ∂X is expressed by the following equation (2). In equation (2), the matrix to which the subscript T is assigned represents the transposed matrix of the original matrix. Therefore, for example, Wa ^T represents a transposed matrix of the matrix Wa.
∂I / ∂X = 2 × {Wa T · (Wa · X-Ya) + Wb T · (Wb · X-Yb)
+ ΓC ^T · C · X} (2)

The gradient ∂I / ∂X based on the matrix X of the high resolution image Fxi is calculated as an update amount for the high resolution image Fxi (where i is a natural number). This calculation process corresponds to the process of step S13 in FIG.

The subtraction unit 64 subtracts the update amount ∂I / ∂X for the high resolution image Fxi from the matrix X of the high resolution image Fxi selected by the selection unit 62 as in step S14 of FIG. A matrix X ′ of the following formula (3) is calculated (where i is a natural number). The matrix X ′ corresponds to a matrix in which pixel values of the high resolution image Fx (i + 1) are written. The high resolution image Fxi is updated by the subtraction processing in the subtraction unit 64, and the updated high resolution image Fx (i + 1) is generated.
X ′ = X−∂I / ∂X (3)

The image data of the high resolution image generated by the update by the subtracting unit 64 is output to the first signal control unit 51 in FIG. During the period when the super-resolution processing is not completed (the simple repetition period and the step display period described later shown in FIG. 9), the image data of the high resolution image output from the subtraction unit 64 is the first signal control unit. It is given to the frame memory 65 via 51. The frame memory 65 temporarily stores the image data of the given high resolution image, and gives this to the selection unit 62. As a result, the high-resolution image output from the subtraction unit 64 is updated again by the update amount calculation unit 63 and the subtraction unit 64. Hereinafter, the process of updating the high-resolution image Fxi only once and obtaining the high-resolution image Fx (i + 1) is referred to as a super-resolution unit process (where i is a natural number).

∙ An upper limit can be set for the number of repetitions of super-resolution unit processing. When the upper limit is set for the number of repetitions of the super-resolution unit process, the super-resolution process is completed when the number of repetitions reaches the upper limit. If it is determined that the update amount for the high-resolution image has become sufficiently small regardless of the number of repetitions of the super-resolution calculation process, the super-resolution process may be completed at that time.

The final high-resolution image obtained after the super-resolution process is completed is called “final high-resolution image”, and the intermediate high-resolution image obtained before the super-resolution process is completed is called “intermediate This is called “resolution image”. For example, when the final high resolution image is the image Fx9, each of the images Fx1 to Fx8 is an intermediate generated high resolution image.

[Control by repetitive control unit 50]
Next, the operation realized by the control of the repetitive control unit 50 will be described in detail. When attention is paid to the control of the repetitive control unit 50, the period around the super-resolution processing based on the images Fa and Fb is classified into a simple repetitive period, a stage display period, and a completion processing period as shown in FIG. . After the images Fa and Fb are given to the super-resolution computing unit 44, a simple repetition period is started, a stage display period is started after the end of the simple repetition period, and a completion processing period is started after the end of the stage display period. Is done. Although the super-resolution processing is completed due to the number of repetitions of the super-resolution unit processing reaching the upper limit number, the completion time is the start time of the completion processing period.

Before the start of the simple repetition period, image data of the image Fa is given to the display processing unit 20 via the signal processing unit 52, and a display image based on the image Fa (corresponding to the image 301 in FIG. 6) is displayed on the display unit 27. Is displayed. Instead of the display image based on the image Fa, a display image based on the image Fb may be displayed. In the following description, when simply referred to as a display image, it refers to an image displayed on the display unit 27, and when simply referred to as a display screen, it refers to a display screen on the display unit 27. In the present embodiment, it is assumed that the display unit 27 is a display unit provided in the imaging device 1, but the display unit 27 is a display device (liquid crystal display or plasma display) external to the imaging device 1. ).

The iterative control unit 50 controls the first and second

signal control units

51 and 53 so that the following operations are executed in the simple repetition period, the stage display period, and the completion processing period.

In the simple repetition period, the image data of the high resolution image output from the super-resolution calculation unit 44 is given only to the super-resolution calculation unit 44 via the first signal control unit 51. Therefore, in the simple repetition period, the display image on the display unit 27 is not updated, while the high-resolution image is updated by repeated execution of the super-resolution unit process.

In the stage display period, the image data of the high resolution image (intermediately generated high resolution image) output from the super resolution calculation unit 44 is supplied to the super resolution calculation unit 44 via the first signal control unit 51 and the first. The signal is supplied to the display processing unit 20 through the 1 signal control unit 51, the signal processing unit 52, and the second signal control unit 53. Therefore, in the stage display period, the display image on the display unit 27 can be updated with the contents of the latest high-resolution image. At this time, the high-resolution image is continuously updated by repeatedly executing the super-resolution unit process.

In the completion processing period, image data of a high resolution image (final high resolution image) output from the super-resolution calculation unit 44 is given only to the signal processing unit 52 via the first signal control unit 51. The video signal of the high resolution image generated by the signal processing unit 52 based on the image data is output to the display processing unit 20 and the compression processing unit 16 via the second signal control unit 53. The completion processing period is a period that comes after completion of the super-resolution processing.

A more specific explanation will be given with numerical examples. As shown in FIG. 9, the start time of the simple repetition period, the start time of the stage display period, and the start time of the completion processing period (that is, the completion time of the super-resolution processing) are t _A , t _B, and t _C , respectively. Suppose there is. Further, it is assumed that the super-resolution processing is completed when the high-resolution image Fx9 is obtained. That is, it is assumed that the image Fx9 is the final high resolution image. Also, assume that the latest high-resolution image at time t _B is Fx3.

The super-resolution computing unit 44 starts super-resolution processing with the time t _A as a starting point (for example, starts generating the initial high-resolution image Fx1). In the simple repetition period, images Fx1, Fx2, and Fx3 are sequentially generated by the super-resolution calculation unit 44. The image data of the images Fx1, Fx2, and Fx3 is supplied from the first signal control unit 51 to the frame memory 65, but is not supplied to the signal processing unit 52. Therefore, the contents of the images Fx1, Fx2, and Fx3 are not reflected in the display image during the simple repetition period.

Thereafter, when reaching the stage display period, between the time _{t B} and time _{t C,} while the image Fx4 ~ FX8 are sequentially generated by the super-resolution computation unit 44, an image Fx4 ~ FX8 first signal controller 51, the signal processing unit 52, and the second signal control unit 53 are sequentially supplied to the display processing unit 20. Therefore, when the image Fx4 is given to the display processing unit 20, the display image of the display unit 27 can be updated using the image Fx4, and when the image Fx5 is given to the display processing unit 20, the image The display image of the display unit 27 can be updated using Fx5. For example, when the image Fx4 is given to the display processing unit 20, the display processing unit 20 displays the whole or a part of the image Fx4 on the display unit 27, and when the image Fx5 is given to the display processing unit 20, the image Fx4 All or part of Fx5 can be displayed on the display unit 27. The same applies to the images Fx6 to Fx8.

Images

303 and 304 in FIG. 6 correspond to images displayed during the stage display period.

It shifts the image Fx9 is generated from the phase display period to complete the treatment period at time t _C. The image data of the image Fx9 is supplied to the signal processing unit 52 via the first signal control unit 51 and converted into a video signal, and the video signal (the video signal of the image Fx9) is transmitted via the second signal control unit 53. To the display processing unit 20 and the compression processing unit 16. When the video signal of the image Fx9 is given, the display processing unit 20 can update the display image of the display unit 27 using the image Fx9. For example, the display processing unit 20 can cause the display unit 27 to display all or part of the image Fx9 when the image Fx9 is given to the display processing unit 20. An image 302 in FIG. 6 corresponds to an image displayed during or after the completion processing period. The compression processing unit 16 compresses the video signal of the image Fx9, and the compressed video signal is stored in the external memory 18.

The iterative control unit 50 determines whether the time point of interest belongs to a simple iterative period or a stage display period based on a predetermined iterative control index, and based on the iterative control index, during the stage display period Controls the update timing of the display image. Examples of this iterative control index are shown below. As will be described later, it is possible to eliminate the simple repetition period. In other words, immediately after the start of the super-resolution processing, the stage display period may be started immediately (time t _A and time t _B may be matched).

The first index for iterative control is the elapsed time from the start time of execution of the super-resolution processing (that is, time t _A ). When the first iterative control index is used, the iterative control unit 50 compares the elapsed time TE from the time t _A at the time of _interest with a predetermined reference elapsed time TE _REF0 . Then, when TE <TE _REF0 , it is determined that the point of interest belongs to the simple repetition period. When TE ≧ TE _REF0 and the super-resolution processing is not completed, the point of _interest becomes the step display period. Judge that it belongs. Then, when the time point of interest belongs to the stage display period, the elapsed time TE is compared with predetermined reference elapsed times TE _REF1 , TE _REF2 , TE _REF3 ,..., And as shown in FIG. When the reference elapsed times TE _REF1 , TE _REF2 , TE _REF3 ,... _Are reached, the first, second, third _,.

Here, 0 <TE _REF0 ≦ TE _REF1 <TE _REF2 <TE _REF3 ,... However, TE _REF = 0 can also be set. If TE _REF is set to zero, there is no simple repetition period.

In the display update process at the point of interest, the latest high-resolution image obtained at the point of interest is reflected in the display image of the display unit 27. For example, when the latest high-resolution image obtained at the time of interest is the image Fx4, the display image is updated so that the whole or a part of the image Fx4 is displayed on the display unit 27. In some cases, the high-resolution image is updated a plurality of times between the first display update process and the second display update process. In this case, the image Fxi is displayed as the display image in the first display update process. In the second display update process, a high-resolution image other than the image Fx (i + 1) (for example, the image Fx (i + 2)) is reflected in the display image.

The second index for iterative control is the number of executions PN of super-resolution unit processing for the images Fa and Fb. When the number of executions PN is 1, 2, 3,..., The images Fx2, Fx3, Fx4,. When the second index for iterative control is used, the iterative control unit 50 compares the number of executions PN at the time of _interest with a predetermined reference number PN _REF0 . If PN <PN _REF0 is satisfied, it is determined that the time point of interest belongs to the simple repetition period. On the other hand, when PN ≧ PN _REF0 is established and the super-resolution processing is not completed, it is determined that the time point of interest belongs to the stage display period. Then, when the time point of interest belongs to the stage display period, the execution number PN is compared with a predetermined reference number PN _REF1 , PN _REF2 , PN _REF3 ,..., And as shown in FIG. When the number of times PN _REF1 , PN _REF2 , PN _REF3 ,... _Is reached, display update processing of the first time, the second time, the third time _,. Here, 1 ≦ PN _REF0 ≦ PN _REF1 <PN _REF2 <PN _REF3,.

The third index for iterative control is the image quality improvement amount of the high resolution image by the repeated execution of the super-resolution unit processing. As described above, the update amount calculation unit 63 in FIG. 8 generates the matrices Wa · X and Wb · X representing the pixel values of the two estimated low-resolution images from the matrix X representing the pixel values of the high-resolution image Fxi. The update amount ∂I / ∂X corresponding to the errors | Wa · X−Ya | and | Wb · X−Yb | between the estimated low resolution image and the actual low resolution image is calculated, and the update amount ∂I / ∂ The high resolution image Fx (i + 1) is generated from the high resolution image Fxi by the update based on X. Such updating with the update amount ∂I / ∂X is repeatedly executed to improve the image quality of the high-resolution image.

Therefore, it is possible to estimate the image quality improvement amount of the high-resolution image by the repeated execution of the super-resolution unit process from the update amount ∂I / ∂X. When using the third index for repetitive control, specifically, the following processing is performed.

The update amount ∂I / ∂X is expressed by a matrix having the same number of elements as the number of elements of the matrix X. The iterative control unit 50 obtains the sum of absolute values of each element of the update amount ∂I / ∂X every time the high-resolution image is updated. The total sum of the absolute values of the update amounts ∂I / ∂X used when generating the high-resolution image Fx (i + 1) from the high-resolution image Fxi is represented by Q _i . The sum Q _i represents the image quality improvement amount of the high resolution image Fx (i + 1) viewed from the high resolution image Fxi, and it can be said that the image quality improvement amount is larger as the sum Q _i is larger.

When the high-resolution image Fx (i + 1) is generated, the iterative control unit 50 calculates the improvement evaluation value EV _i according to the expression “EV _i = (Q ₁ + Q ₂ +... Q _i )”. This improvement evaluation value EV _i represents the image quality improvement amount of the high resolution image Fx (i + 1) viewed from the initial high resolution image Fx1. For this reason, it can also be considered that an iterative control unit 50 includes an estimation unit (not shown) that estimates the image quality improvement amount by calculating the improvement evaluation value EV _i . FIG. 12 shows the relationship between the number of executions PN of the super-resolution unit process and the improvement evaluation value EV _i .

The iterative control unit 50 using the third index for iterative control compares the improvement evaluation value EV _i at the time of _interest with a predetermined reference evaluation value EV _REF0 . When EV _i <EV _REF0 , it is determined that the point of interest belongs to the simple repetition period, and when EV _i ≧ EV _REF0 and super-resolution processing is not completed, the point of _interest is displayed in stages. Judge as belonging to the period. Then, when the time point of interest belongs to the stage display period, the improvement evaluation value EV _i is compared with predetermined reference evaluation values EV _REF1 , EV _REF2 , EV _REF3 ,... When the value EV _i reaches the reference reference evaluation values EV _REF1 , EV _REF2 , EV _REF3 ,..., Display update processing of the first time, the second time, the third time _,. Here, 0 <EV _REF0 ≦ EV _REF1 <EV _REF2 <EV _REF3 .

[Display processing section]
Next, a configuration example of the display processing unit 20 will be shown. FIG. 13 is an internal block diagram of the display processing unit 20 according to the first embodiment. The display processing unit 20 in FIG. 13 includes a display video processing unit 71, a VRAM (Video Random Access Memory) 72, and a display driver 73.

In the shooting mode, the video signal from the video signal processing unit 13 is input to the display video processing unit 71, and in the playback mode, the video signal from the external memory 18 is input to the display video processing unit 71 via the expansion processing unit 19. Entered. The video signal input to the display video processing unit 71 is, for example, a video signal of a low resolution image that has not been subjected to super-resolution processing, or a final high resolution image output from the video signal processing unit 13 of FIG. Alternatively, it is a video signal of an intermediate generated high resolution image.

The display video processing unit 71 converts the resolution of the image so that the image (low-resolution image or high-resolution image) represented by the given video signal can be displayed on the display unit 27, and the resolution The video signal of the converted image is written into the VRAM 72. The VRAM 72 is a video display memory for the display unit 27. The display driver 73 displays an image represented by the video signal written in the VRAM 72 on the display screen of the display unit 27.

[Video processor for display]
Although it is possible to display the entire high-resolution image (and low-resolution image) on the display screen of the display unit 27, the resolution of the display panel 27 provided in the imaging device 1 is normally set to the video signal processing unit 13. Therefore, if the entire image area of the high-resolution image is displayed on the display screen, the effect of the super-resolution processing is difficult to understand. Considering this, a partial image of the high resolution image instead of the entire high resolution image may be displayed on the display unit 27. Of course, when the display unit 27 is a display device external to the imaging device 1 and the display device has a sufficiently high resolution, it may be displayed on the entire high-resolution image. Hereinafter, a processing example in the case of displaying a partial image of the high resolution image on the display unit 27 will be described.

The display video signal processing unit 71 extracts a partial image area of the high-resolution image as a cut-out area by using a cut-out process of cutting out a part from the entire high-resolution image. If necessary, a reduction process for reducing the image size of the high-resolution image can be performed before and after the clipping process. The image size after the cutout area or the image size after the reduction process and the cutout area is determined according to the resolution of the display unit 27. In FIG. 14A, reference numeral 320 represents the entire image area of the high resolution image, and reference numeral 321 represents the cutout area. A reference numeral 322 in FIG. 14B represents a display image corresponding to an image in the cutout area 321.

Simply, for example, a region having a predetermined shape at a predetermined position on the high-resolution image can be set as a cut-out region. More specifically, for example, a rectangular area having a predetermined image size located near the center of the high-resolution image can be extracted as a cut-out area.

It is also possible to set a face area where a face portion of a person appears or an area including a face area as a cutout area. A face detection unit (not shown) included in the video signal processing unit 13 is, for example, based on image data of a low-resolution image Fa (or Fb) that is a source of a high-resolution image, by a known face detection process, The position and size of the face area on (or Fb) is detected. Based on this detection result, the position and size of the face region on the high resolution image can be calculated, and the position and size of the cutout region on the high resolution image can be obtained from the calculation result.

Also, the focused area can be set as the cutout area. This is because there is a high possibility that the main subject that the photographer pays attention exists in the in-focus area. For example, an in-focus area can be detected based on an AF evaluation value used for autofocus control using a TTL (Through The Lens) type contrast detection method. More specifically, for example, an AF evaluation unit (not shown) included in the video signal processing unit 13 converts the entire image area of the low-resolution image Fa (or Fb) that is the source of the high-resolution image into a plurality of AF evaluation areas. The image is divided and the contrast in each AF evaluation area is detected from the image data of the low-resolution image Fa (or Fb). By calculating the amount of the high frequency component included in the AF evaluation area, an AF evaluation value corresponding to the contrast can be obtained. Then, the AF evaluation area having the largest detected contrast (AF evaluation value) among the plurality of AF evaluation areas is determined to be an in-focus area, and a high level corresponding to the in-focus area is determined. An area on the resolution image is set as a cutout area.

Further, the position and size of the cutout area may be set according to a manual operation by the user (such as an operation on the operation unit 26).

[Display additional items]
It is also possible to superimpose additional items on an image representing the whole or part of a high-resolution image and display the image after superimposition. Examples of additional items (

indexes

331, 332, and 340 and an image 335) will be described below. The processing for generating the additional item can be performed by the display video processing unit 71 or can be performed by the video signal processing unit 13.

For example, as shown in FIG. 15, in a simple repetition period or a stage display period, an index 331 representing the remaining processing time and the elapsed time TE is superimposed on an image representing a part (or the whole) of the high-resolution image, and the superimposed image Is displayed as a display image. The index 331 is a rectangular area having a longitudinal direction with respect to the horizontal direction of the image and having a first color and a second color, and the remaining processing is performed according to the area ratio of the first color and the second color area occupying the rectangular area. Express time and elapsed time TE. In FIG. 15, the first color area is indicated by a hatched area. The first and second color areas are respectively arranged on the left and right sides of the rectangular area representing the entire index 331, and coincide with the left end of the second color area as the elapsed time TE increases. The right end moves to the right. The remaining processing time at the point of interest is the time required to complete the super-resolution processing calculated from the point of interest, and is obtained by subtracting the elapsed time TE at the point of interest from the total processing time. The total processing time is the time from time t _A to time t _C in FIG. 9, and the number of times of super-resolution unit processing to be repeatedly executed until the super-resolution processing is completed, the low-resolution image, and the high-resolution image. It is calculated from the image size.

Further, for example, as shown in FIG. 16, in the simple repetition period or the stage display period, an index 332 indicating the execution timing of the display update process is superimposed on the image representing a part (or the whole) of the high-resolution image together with the index 331, The superposed image may be displayed as a display image. In FIG. 16, the index 332 is formed from a plurality of line segments located below the index 331. As the elapsed time TE increases, the right edge of the first color area moves to the right, and when the position of the right edge in the left-right direction matches one of the drawing positions of the plurality of line segments, the display update process is 1 It is executed batchwise. The drawing position of each line segment is determined by setting or predicting the execution timing of the display update process in advance.

Also, for example, as shown in FIG. 17, in the stage display period or completion processing period, the entire reduced image 335 of the high resolution image is superimposed on an image representing a part of the high resolution image (image of the cutout region), and after the superimposition These images may be displayed as a display image. The reduced image 335 may be an entire reduced image of the low resolution image (Fa or Fb) that is the source of the high resolution image.

Further, for example, as shown in FIG. 18A, in the stage display period or the completion processing period, an index 340 indicating the degree of expectation of the effect of the super-resolution processing is superimposed on an image representing a part (or the whole) of the high resolution image. Then, the superimposed image may be displayed as a display image. More specifically, for example, using the icons as shown in FIG. 18B, the expectation is classified and displayed in three stages. The resolution (substantial resolution) of the high-resolution image obtained by the super-resolution processing is higher than that of the low-resolution image. The degree of expectation here refers to the degree of improvement in resolving power of a high-resolution image obtained by super-resolution processing with respect to a low-resolution image.

The resolution improvement by the super-resolution processing is realized on the premise that there is a positional shift in units of subpixels between the images Fa and Fb. If the amount of motion between the images Fa and Fb is completely zero, the image Fa and the image Fb are completely the same image (ignoring the motion of the subject in the real space), so that the resolution is improved by super-resolution processing. Can't hope. On the other hand, if there is an appropriate misalignment between the images Fa and Fb, high resolution improvement can be expected.

From such a viewpoint, the degree of expectation can be estimated based on the amount of motion between the low-resolution images Fa and Fb that are the basis of the high-resolution image. For example, the horizontal component and the vertical component of the motion amount of the images Fa and Fb are detected based on the stored contents of the motion amount storage unit 43, and further, the decimal point part M _abH of the detected horizontal component and the decimal point part M of the detected vertical component are detected. _{Obtain abV} . The decimal point part here is a decimal point part when the adjacent pixel interval pp _L of the low resolution image is set to 1. Then, an evaluation value EV _A is obtained based on the decimal point parts M _abH and M _abV , and when EV _A > EV _A1 , the expectation is estimated to be the first expectation, and EV _A1 ≧ EV _A > EV _A2 In some cases, the degree of expectation is estimated to be the second degree of expectation. When EV _A2 ≧ EV _A , the degree of expectation is estimated to be the third degree of expectation, and the estimated degree of expectation is reflected in the indicator 340. . The values of EV _A1 and EV _A2 are set in advance so that “EV _A1 > EV ₂ > 0” is satisfied.

Here, the evaluation value EV _A is an evaluation value that takes a higher value as the decimal point portions M _abH and M _abV are _closer to (0.5 × pp _L ). For example, the reciprocal of (| M _abH −0.5 × pp _L | + | M _abV −0.5 × pp _L |) is set as the evaluation value EV _A. In the super-resolution processing, the positional deviation between the images Fa and Fb corresponding to the amount of motion between the images Fa and Fb is corrected, and the images Fa and Fb after the positional deviation correction are combined to achieve high resolution. If both M _abH and M _abV are zero (or substantially zero), as shown in FIG. 19A, the corresponding pixel positions of the images Fa and Fb after the positional deviation correction are overlapped, and the correspondence between the images Fa and Fb. The pixel only shows information on the same position of the subject. In such a situation and similar situations, a high resolution improvement effect cannot be expected. On the other hand, if the decimal points M _abH and M _abV are close to (0.5 × pp _L ), as shown in FIG. 19B, the vicinity of the intermediate position between adjacent pixels of the image Fa, which was not obtained with the image Fa alone, is obtained. Since the information on the subject in can be obtained from the image Fb, a high resolution improvement effect can be expected.

[Execute super-resolution processing during playback]
The above description assumes that super-resolution processing is performed in the shooting mode, but it is also possible to perform super-resolution processing in the reproduction mode. In this case, first, the low resolution images Fa and Fb are acquired in the shooting mode, and the image data of the images Fa and Fb are compressed by the compression processing unit 16 and then recorded in the external memory 18. Thereafter, in the reproduction mode, the compressed image data of the images Fa and Fb recorded in the external memory 18 are expanded by the expansion processing unit 19, and the image data of the images Fa and Fb obtained by the expansion are sequentially shown in FIG. What is necessary is just to input into the super-resolution processing part 40 shown.

That is, the input subject of the image data of the images Fa and Fb to the super-resolution processing unit 40 is the AFE 12 when super-resolution processing is performed in the shooting mode, whereas the super-resolution processing is performed in the reproduction mode. In some cases, it is the decompression processing unit 19. Except that the input subject of the image data of the images Fa and Fb to the super-resolution processing unit 40 is different, the part that receives the operation of the super-resolution processing unit 40 and the output data of the super-resolution processing unit 40 (repetition control unit 50, The operations of the first signal control unit 51, the signal processing unit 52, the second signal control unit 53, and the display processing unit 20) are the same between the shooting mode and the reproduction mode. Therefore, even when super-resolution processing is performed in the reproduction mode, display update processing using an intermediate high-resolution image is executed as described above.

When the super-resolution processing based on the images Fa and Fb is completed and the final high-resolution image based on the images Fa and Fb is obtained, the image data of the final high-resolution image is transferred to the external memory 18 via the compression processing unit 16. To be recorded. Once the final high-resolution image based on the images Fa and Fb is obtained, it is not necessary to re-execute the super-resolution processing based on the images Fa and Fb.

Further, the example in which the super-resolution processing unit 50 is provided in the video signal processing unit 13 has been described (see FIG. 7). However, the super-resolution processing unit 50 and the super-resolution processing unit 50 and It is also possible to provide a part (mainly the repetition control unit 50) responsible for execution control of the display update process in the display processing unit 20.

<< Second Example >>
A second embodiment of the present invention will be described. In the first embodiment described above, the entire high resolution image Fx1 is generated from the entire low resolution images Fa and Fb, and then the high resolution image Fx (i + 1) is generated from the entire high resolution image Fxi by one super-resolution unit process. ) Is generated at the same time. In consideration of the fact that it takes a considerable amount of time to obtain the final high-resolution image, an intermediate high-resolution image that should be called an intermediate result of the super-resolution processing is displayed during the super-resolution processing. Yes.

On the other hand, in the second embodiment, the entire image area of the low resolution image and the high resolution image is divided into a plurality of areas, and the super-resolution processing for the plurality of divided areas is executed in a time division manner. Then, the results of the super-resolution processing for the divided areas are displayed in order from the divided areas for which the super-resolution processing has been completed.

More specific explanation. FIG. 20 shows an internal block diagram of the super-resolution processor 40a according to the second embodiment. The super-resolution processing unit 40a can be provided in the video signal processing unit 13 or the display processing unit 20 in FIG. The super-resolution processing unit 40a includes each part referred to by reference numerals 41 to 43 and 44a. The memory unit 41, the motion amount calculation unit 42, and the motion amount storage unit 43 in the super-resolution processing unit 40a are the same as those shown in FIG.

In the playback mode, the image data of the images Fa and Fb are input to the super-resolution processing unit 40a from the external memory 18 via the expansion processing unit 19 and in the shooting mode from the AFE 12. As described in the first embodiment, the motion amount between the images Fa and Fb is calculated by the motion calculation unit 42 and stored in the motion amount storage unit 43. The image data of the images Fa and Fb are input to the super-resolution calculation unit 44a via the memory unit 41.

The super-resolution calculation unit 44a has the same configuration and function as the super-resolution calculation unit 44 of FIG. However, in the super-resolution calculation unit 44a, the output data of the subtraction unit 64 is directly given to the frame memory 65, and the frame memory 65 stores the image data of the high resolution image output from the subtraction unit 64. 8 generates the entire image data of the high-resolution image at the same time, the super-resolution calculation unit 44a has the first, second, and second in the high-resolution image. The image data of the divided areas 3,.

For the sake of concrete explanation, let us consider a case where the entire image area of the low resolution image and the high resolution image is divided into nine. As shown in FIG. 21, the divided image areas DR ₁ to DR ₉ are formed by dividing the entire image areas of the low resolution images Fa and Fb and the high resolution image into three equal parts in the vertical and horizontal directions, respectively. The entire image area of the image Fa is a combination of the divided areas DR ₁ to DR ₉ of the image Fa, and the entire image area of the image Fb is a combination of the divided areas DR ₁ to DR ₉ of the image Fb. The entire image area of the high resolution image is a combination of the divided areas DR ₁ to DR ₉ of the high resolution image.

The super-resolution calculation unit 44a performs super-resolution processing for each divided region. The contents of the individual super-resolution processing are the same as those shown in the first embodiment. The super-resolution calculation unit 44a performs an operation of “performing the super-resolution processing for the divided region DR _j and executing the super-resolution processing for the divided region DR _{j + 1} after the completion” for the super-resolution for all the divided regions. Repeat until image processing is complete (where j is a natural number).

With reference to FIGS. 22 and 23, the state of super-resolution processing and display image transition according to the second embodiment will be described. At time _{t 0} the image data of the image Fa and Fb are input to the super-resolution calculating unit 44a, as shown in FIG. 22, after time _{t 0,} as time progresses, the time _t _1, t 2, · · · , T ₉ are visited in this order. FIG. 23 shows how the display image changes over time. In FIG. 23,

images

401, 402, 403, and 404 indicate display images at times t ₀ , t ₁ , t ₂ , and t ₉ , respectively, and hatched areas shown in the display image of FIG. FIG. 4 shows a part after super-resolution processing.

At time t ₀ , the display processing unit 20 displays the entire image of the low resolution image Fa as the display image 401. The super-resolution calculation unit 44a performs super-resolution processing using the image data in the divided region DR ₁ of the image Fa and the image data in the divided region DR ₁ of the image Fb between times t _{0 and} t _1. , thereby generating image data of the divided area DR ₁ of the high-resolution image. Thereafter, during the time t ₁ -t ₂ , super-resolution processing is executed using the image data in the divided area DR ₂ of the image Fa and the image data in the divided area DR ₂ of the image Fb, and thereby the high-resolution image generating image data of the divided area DR _2. The same applies to the time between t _{2 and} t ₃ . That is, the super-resolution computation unit 44a, between the time _{t _j-1} -t _j, super-resolution image using the image data of the divided region DR _j of the image data and the image Fb of the divided region DR _j image Fa The processing is executed, thereby generating image data in the divided region DR _j of the high resolution image (where j is a natural number). The super-resolution operation unit 44a sequentially executes such unit processing for the time t _j−1 -t _j for nine times.

Super-resolution processing using the image data in the divided region DR _j of the image data and the image Fb of the divided region DR _j image Fa is executed based on the amount of motion between images Fa and Fb of the divided regions DR _j Is done. The motion amount between the images Fa and Fb for the divided region DR _j is calculated by the motion amount calculation unit 42 and stored in the motion amount storage unit 43. Motion amount calculation unit 42, based on the image data in the divided region DR _j of the image data and the image Fb of the divided region DR _j image Fa, the movement amount between images Fa and Fb of the divided regions DR _j calculate. That is, the motion amount calculation unit 42 calculates the motion amount between the images Fa and Fb for each divided region. However, one motion amount may be obtained for the entire images Fa and Fb, and the one motion amount may be commonly used as the motion amount for all the divided regions DR ₁ to DR ₉ between the images Fa and Fb. Is possible.

At time t _1, when the image data of the divided area DR ₁ of the high-resolution image is generated, the image data is sent to the display processing unit 20 from the super-resolution processing section 40a, the display processing unit 20 is a low resolution Based on the image data in the divided regions DR ₂ to DR ₉ of the image Fa and the image data in the divided region DR ₁ of the high resolution image, the display image 402 of FIG. The display image 402 is an image obtained by synthesizing an image in the divided regions DR ₂ to DR ₉ of the low resolution image Fa and an image (intermediately generated image) in the divided region DR ₁ of the high resolution image.

Similarly, at time t _2, the the image data of the divided area DR ₂ of the high-resolution image is generated, the image data is sent to the display processing unit 20 from the super-resolution processing section 40a, the display processing unit 20 The display image 403 of FIG. 23 is displayed on the display unit 27 based on the image data in the divided areas DR ₃ to DR _{9 of} the low resolution image Fa and the image data in the divided areas DR ₁ and DR ₂ of the high resolution image. Let The display image 403 is an image obtained by synthesizing the images in the divided regions DR ₃ to DR ₉ of the low resolution image Fa and the images (two intermediate generation images) in the divided regions DR ₁ and DR ₂ of the high resolution image. is there.

A similar display image update is performed at times t ₃ to t ₈ , and when image data in the divided region DR ₉ of the high-resolution image is generated at time t ₉ , the entire image regions of the images Fa and Fb are updated. The super-resolution process ends. At time t ₉ , the image data in the high-resolution image divided region DR ₉ is sent to the display processing unit 20, and the display processing unit 20 sends the divided region of the high-resolution image sent from time t ₁ to t ₉ . A display image 404 of FIG. 23 is displayed on the display unit 27 based on the image data in DR ₁ to DR ₉ . The display image 404 is an image obtained by synthesizing images in the divided regions DR ₁ to DR ₉ of the high resolution image, that is, the entire image of the high resolution image to be finally generated. All image data of the high resolution image generated by the super-resolution processing unit 40 a is recorded in the external memory 18 via the compression processing unit 16.

In the above description, the divided area to be subjected to the super-resolution processing is scanned like raster scanning from the upper left to the lower right, but this scanning method can be arbitrarily changed.

<< Third Example >>
A third embodiment of the present invention will be described. In the first and second embodiments described above, it is assumed that the arithmetic processing (image processing) to be performed on the input image is a super-resolution processing, and a time corresponding to the generation of the output image to be finally obtained. In view of this, intermediate results of arithmetic processing generated in the execution process of arithmetic processing are displayed step by step. The input images for the first and second embodiments are a plurality of low-resolution images, and the output images for the first and second embodiments are one final high-resolution image.

However, in the present invention, the predetermined arithmetic processing for generating the output image from the input image is not limited to the super-resolution processing, and processing such as the super-resolution unit processing of the first embodiment is included in the arithmetic processing. It does not matter whether or not iterative execution is included. Therefore, the number of input images for obtaining an output image is not limited to a plurality, and may be one. The predetermined arithmetic processing is arbitrary image processing such as spatial filtering, frequency filtering, and geometric transformation. As described above, the display method according to the present invention can be applied to any apparatus or method that generates an output image from an input image by a calculation process that takes a relatively long time (for example, several seconds to several tens of seconds). Of course, the input image and the output image are different from each other.

The input image may not be an image obtained by photographing with the imaging device 1, but now it is assumed that the input image is a single frame image obtained by photographing with the imaging device 1. A specific example of the display method according to the third embodiment will be described. A specific example of this display method is similar to that according to the second embodiment.

Video signal processing unit 13 generates an output image I _O of one by performing predetermined arithmetic processing for one input image (frame image) I _IN. The video signal processing unit 13 divides the entire image area of the input image I _IN and the output image _IO into a plurality of divided areas. For the sake of concrete explanation, as shown in FIG. 21, the divided image areas DR ₁ to DR ₉ are divided by dividing the entire image areas of the input image I _IN and the output image _IO into three equal parts in the vertical and horizontal directions, respectively. Form. The entire image region of the input image _{I IN} is a composite of the divided regions _DR 1 ~ DR ₉ of the input image _{I IN,} the entire image area of the output image _{I O} is the output image _{I O} of the divided regions _DR 1 ~ DR ₉ is synthesized. The video signal processing unit 13 executes predetermined arithmetic processing for each divided region. Video signal processing section 13, to "perform a predetermined calculation process on the image in the divided region DR _j of the input image I _IN, the image of the divided region DR j _{+ 1} of the input image I _IN after it has been completed The predetermined operation processing is repeatedly executed until the predetermined operation processing is completed for all the divided areas (where j is a natural number).

As time progresses after time t ₀ , time t ₁ , t ₂ ,..., T ₉ come in this order. At time _{t 0,} the display processing unit 20 displays the entire image of the input image _{I IN} as the display image. The video signal processing unit 13 performs predetermined arithmetic processing on the image (image data) in the divided region DR _j of the input image I _IN between the times t _{j−1 and} t _j , and thereby the output image I An image (image data) in the divided region DR _{j of} _O is generated (where j is a natural number). Such unit processing between time t _{j−1 and} t _j is sequentially executed nine times.

At time t _1, when the image data of the divided area DR ₁ of the output image I _O is generated, the image data is sent to the display processing unit 20, the display processing unit 20, the divided region DR of the input image I _IN _{Based on} the image data in ₂ to DR ₉ and the image data in the divided area DR ₁ of the output image I _O , the image in the divided areas DR ₂ to DR ₉ of the input image I _{IN and} the divided area DR of the output image I _O An image obtained by synthesizing the image in ₁ (intermediately generated image) is displayed on the display unit 27.

Similarly, at time t _2, the the image data of the divided area DR ₂ of the output image I _O is generated, the image data is sent to the display processing unit 20, the display processing unit 20, the input image I _IN Based on the image data in the divided regions DR ₃ to DR ₉ and the image data in the divided regions DR ₁ and DR ₂ of the output image _IO , the images and output images in the divided regions DR ₃ to DR ₉ of the input image I _IN An image obtained by synthesizing the images (two intermediately generated images) in the _IO divided regions DR ₁ and DR ₂ is displayed on the display unit 27.

A similar display image update is performed at times t ₃ to t ₈ , and when image data in the divided region DR ₉ of the output image _IO is generated at time t ₉ , the output image I _{IN is} output from the input image I _IN. The process for generating _O ends. At time _{t 9,} the image data in the divided region DR ₉ of the output image _{I O} is sent to the display processing unit 20, the display processing unit 20 divides the output image _{I O} sent at time _t 1 ~ _{t 9} based on the image data in the region _DR 1 ~ DR _9, the displayed image, i.e., the entire image of the output image _{I O} to be finally generated synthesized image of the divided region _DR within 1 ~ DR ₉ of the output image _{I O} This is displayed on the unit 27. All the generated image data of the output image _IO is recorded in the external memory 18 via the compression processing unit 16.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 5 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above-described embodiment, the number of low-resolution images used for generating a high-resolution image is two, but it may be other than two.

[Note 2]
In the above description, the amount of motion between real low-resolution images is derived by computation based on image data, but based on the detection result of a sensor (not shown) that detects the motion of the imaging device 1 in real space, You may make it derive | lead-out the motion amount between real low-resolution images. The sensor that detects the movement of the imaging device 1 is, for example, an angular velocity sensor that detects an angular velocity of the imaging device 1, an angular acceleration sensor that detects angular acceleration of the imaging device 1, or an acceleration sensor that detects acceleration of the imaging device 1, Or a combination thereof. Further, the amount of motion between actual low-resolution images may be derived based on both the detection result of such a sensor and the image data.

[Note 3]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, all or part of the processing executed in the video signal processing unit 13 and the display processing unit 20 can be realized using software. Of course, it is also possible to form them only by hardware. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 4]
For example, it can be considered as follows. In the first embodiment, the super-resolution processing unit 40, the iteration control unit 50, the first signal control unit 51, the signal processing unit 52, the second signal control unit 53 of FIG. 7 and the display processing unit 20 of FIG. The first image display device is formed by the included block. It can be considered that the display unit 27 is further included in the first image display device. The first image display device includes a super-resolution processing unit 40 that generates a high-resolution image from a plurality of low-resolution images by super-resolution processing, and a display based on the high-resolution image generated by the super-resolution processing unit 40 A display control unit that displays an image on the display unit 27. The display control unit is mainly formed by the iterative control unit 50 and the display processing unit 20, and the display control unit includes all or one of the first signal control unit 51, the signal processing unit 52, and the second signal control unit 53. It can be considered that the part is included.

In the second embodiment, a super-resolution processing unit 40a (FIG. 20) that generates a high-resolution image from a plurality of low-resolution images by super-resolution processing, and a high-resolution image generated by the super-resolution processing unit 40a. A second image display device is formed by a block including the display processing unit 20 (FIG. 13) that displays a display image based on the display unit 27. It can be considered that the display unit 27 is further included in the second image display device.

In the third embodiment, a third image display device is formed by blocks including the video signal processing 13 and the display processing unit 20. It can be considered that the display unit 27 is further included in the third image display device.

[Note 5]
The functions of the first, second, or third image display device described above can be realized by an electronic device (for example, an image reproduction device having an image processing function; not shown) different from that of the imaging device 1. In this case, an image display device equivalent to the first, second, or third image display device is provided in the electronic device, and one or more frame images are acquired by the imaging device 1, and then one of the images is acquired. The image data of the frame image described above may be supplied to the electronic device wirelessly or by wire or via a recording medium.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Video signal processing part 20 Display processing part 27 Display part 33

Imaging element

40, 40a

Super-resolution processing part

44, 44a Super-resolution calculating part 50 Iteration control part

Claims

An arithmetic processing unit that generates an output image from an input image by a predetermined arithmetic process;
A display control unit that causes a display unit to display a display image based on a generated image of the arithmetic processing unit;
The display control unit causes the display unit to display an intermediate result of the calculation process generated in the execution process of the calculation process step by step during the execution of the calculation process.
The arithmetic processing includes unit processing that is repeatedly executed,
The arithmetic processing unit updates the intermediate generation image by repeatedly executing the unit processing on the intermediate generation image based on the input image, and finally generates the output image,
2. The display control unit according to claim 1, wherein when the unit process is repeatedly executed, the display control unit generates a display image from the intermediate generation image in the process of repeating the unit process and displays the display image on the display unit. The image display device described.
When the unit process is repeatedly executed, the display control unit displays the display content of the display step by step according to the elapsed time from the execution start time of the arithmetic process or the number of times the unit process is repeatedly executed. The image display apparatus according to claim 2, wherein the latest intermediate generation image at the time of update is reflected in the display content.
The arithmetic processing unit sequentially updates the intermediate generation image by repeatedly executing the unit processing in order to improve the image quality of the intermediate generation image,
The image display apparatus further includes an estimation unit that estimates an image quality improvement amount of the intermediate generated image by the repeated execution of the unit process,
The display control unit updates the display content of the display step by step according to the estimated image quality improvement amount when the unit process is repeatedly executed, and the latest intermediate generation image at the time of update is used as the display content. The image display device according to claim 2, which is reflected.
The calculation process includes first to n-th unit processes,
An i-th intermediate generation image based on a part of the input image is generated as a part of the output image by the i-th unit process.
The entire output image is formed by synthesizing the first to n-th intermediate generation images generated by the first to n-th unit processes.
The display control unit generates a display image using the first to m-th intermediate generation images obtained at that time during execution of the arithmetic processing and causes the display unit to display the display image.
2. The image display apparatus according to claim 1, wherein n and m are natural numbers, n> m is established, and i is an integer satisfying 1 ≦ i ≦ n.
An imaging unit for acquiring an image by shooting;
An image display device,
The image display device according to any one of claims 1 to 5 is used as the image display device.
The image display device receives the image acquired by the imaging unit as the input image.