KR20070032216A - Imaging method and imaging apparatus - Google Patents

Abstract

The present invention is for preventing exposure blurring and satisfactory S / N ratio at the time of imaging low-light subjects. With respect to the first image of a plurality of captured images, image stabilization is set to OFF in S11. In S12, the filter coefficient k is set to one. Initial time is control to eliminate the transient response of the filter. In S13, 1 is added to the counter. Then, the process ends. In the processing on the next image, camera shake is detected and corrected in S20. That is, according to the detected motion vector, correction is performed to eliminate the jitter component. In S21, the coefficient k of the filter is calculated according to the reliability R and the filter coefficient Ky. In S13, 1 is added to the counter and the process ends. The calculated filter coefficient k is supplied to the filter, and the filter coefficient is set to an appropriate value.

Imaging element, shake detection unit, motion vector detection unit, feature extraction unit

Description

Imaging method and imaging device {IMAGING METHOD AND IMAGING APPARATUS}

BRIEF DESCRIPTION OF THE DRAWINGS The block diagram which shows the whole structure of one Embodiment of this invention.

2 is a block diagram of an example of a camera shake detection unit according to one embodiment of the present invention;

3 is a schematic diagram for explaining motion detection in one embodiment of the present invention.

Fig. 4 is a schematic diagram showing three-dimensionally the relationship between the evaluation value and the shift at the time of motion vector detection.

Fig. 5 is a schematic diagram showing two-dimensionally the relationship between the evaluation value and the shift at the time of motion vector detection.

6 is a block diagram showing an example of a filter in one embodiment of the present invention;

7 is a flowchart illustrating a flow of a process of an embodiment of the present invention.

Fig. 8 is a schematic diagram for explaining the determination processing of the reliability in one embodiment of the present invention.

9 is a schematic diagram for explaining a block in another embodiment of the present invention;

Fig. 10 is a schematic diagram showing two-dimensionally the relationship between the evaluation value and the shift at the time of motion vector detection in another embodiment of the present invention.

(Explanation of symbols for the main parts of the drawing)

120: imaging element 140: camera shake detection unit

141: motion vector detection unit 142: feature extraction unit

150: memory controller 151: image memory

160: filter 170: system controller

180: luminance detector 220: representative point extraction circuit

230: subtractor 250: motion vector detection circuit

301 detection area 302 representative point

304: motion vector

401: relationship between evaluation value and deviation during normal hand shake

402: relationship between evaluation value and shift at low contrast

403: Relationship between the evaluation value and the shift when entering the moving subject

The present invention relates to an image pick-up method and an image pick-up device capable of obtaining still images, moving pictures, and the like having good image quality.

In the imaging device, in the low light state, since the amount of incident light is small, the amount of charge accumulation decreases, and the random noise component such as shot noise of the signal output from the imaging element is relatively high, so that the S / N ratio worsens. S / N ratio can be improved by increasing charge accumulation amount by performing exposure for a long time and making a random component relatively low. However, since the exposure for a long time causes blur of the image (it is appropriately referred to as exposure blur) due to camera shake during exposure, it is necessary to fix the imaging device by a tripod or the like.

In order to solve such a problem, Patent Document 1 described below repeats imaging at a shutter speed at which exposure blur is hardly generated, for example, 1/30 second, and corrects camera shake of a plurality of images obtained by this continuous shooting. Polymerization is described. The jitter here is a jitter that occurs between the intervals of the exposure period, for example, between one field and one frame. Similarly, Patent Document 2 describes dividing all the exposure time into a plurality of exposure periods, and adding an image obtained in each exposure period by image stabilization to add a good image with good image quality.

Patent Document 1: Japanese Patent Application Laid-Open No. 9-261526

Patent Document 2: Japanese Patent Application Laid-Open No. 11-75105

As described in Patent Literature 1 or Patent Literature 2, since the method of adding a plurality of images for which image stabilization is added is a simple addition, since the memory capacity increases according to the number of images to be added, images that can be added There is a problem that the number of sheets is limited, and sufficient improvement in S / N ratio cannot be achieved. In addition, in an image into which an unnecessary moving subject or the like enters and an image having low contrast, the accuracy of the motion vector for image stabilization is lowered, resulting in insufficient improvement in image quality.

It is therefore an object of the present invention to provide an image pickup method and an image pickup device capable of generating a captured image with good image quality without causing camera shake even in a low light condition.

(Means to solve the task)

MEANS TO SOLVE THE PROBLEM In order to solve the above-mentioned subject, this invention provides the image capture step of capturing the several different image temporally, the motion detection step which detects a movement from a some image, and produces | generates motion information, and the reliability of the motion information. A reliability determination step for determining, a camera shake correction step for correcting camera shake occurring between the plurality of images in accordance with the motion information, and a filter step for filtering the image after the camera shake correction by the cyclic filter, and the reliability of the movement information. It is an imaging method which changes the characteristic of a filter process accordingly.

The present invention includes image capturing means for capturing a plurality of different images in time, motion detecting means for detecting motion from the plurality of images, generating motion information, reliability determining means for determining the reliability of motion information, and motion. Image stabilization means for compensating for image blur between a plurality of images in accordance with the information, and filter means for filtering the image after image stabilization by a cyclic filter, and for changing the characteristics of the filter means in accordance with the reliability of the motion information. It is an imaging device.

EMBODIMENT OF THE INVENTION Hereinafter, the imaging device which concerns on one Embodiment of this invention is demonstrated, referring drawings. The overall configuration of one embodiment is shown in FIG. 1. In FIG. 1, reference numeral 110 denotes an imaging optical system. The imaging optical system 110 includes a zoom lens for enlarging / reducing a subject, a focus lens for adjusting a focal length, an iris (iris) for adjusting the amount of light, a ND (Neutral Density) filter, and lenses and iris thereof It has a drive circuit for driving. The zoom lens, the focus lens, the IRIS, and the ND filter are driven by the driver 111.

The subject light that has passed through the imaging optical system 110 is incident on the imaging device 120 using a device such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like, and the imaging device 120 according to the subject light An imaging signal is captured. Moreover, although an example of an imaging device is a digital camera, it may be a PDA (Personal Digital Assistants), a mobile phone, etc., and the apparatus which photographs a moving image may be sufficient as it.

The imaging device 120 may be either a primary color system or a complementary color system, and the primary color signal or complementary color system of each RGB color photoelectrically converts a subject incident on the optical imaging system 110. Outputs an analog signal. Various timing signals are supplied from the timing generator (denoted by TG in the drawing) 121 to the imaging device 120, and the imaging device 120 is driven. The timing generator 121 generates various timing signals for driving the imaging device 120.

The imaging signal from the imaging device 120 is supplied to an analog signal processing circuit 130 configured as an integrated circuit (IC). In the analog signal processing unit 130, the sample is held for each color signal, the gain is controlled by AGC (Automatic Gain Control), and an imaging signal which is converted into a digital signal is output by A / D conversion.

The digital imaging signal from the analog signal processor 130 is supplied to the memory controller 150, the camera shake detector 140, and the luminance detector 180. The motion detection unit 140 detects each motion of the plurality of captured images and outputs a motion vector as motion information. It is composed of a motion vector detector 141 and a feature extractor 142. The motion vector detector 141 detects a motion vector based on a time series digital image signal output from the analog signal processor 130. The feature extraction unit 142 extracts feature information. The feature information is an evaluation value corresponding to the detected motion vector. The luminance detector 180 detects a luminance level of a signal output from the analog signal processor 130.

The detected motion vector, the extracted feature information, and the detected luminance level are supplied to the system controller 170. The system controller 170 calculates the reliability of the detected motion vector based on the feature information and the luminance level.

The memory controller 150 is a part that controls the image memory 151. The image memory 151 is a memory for adjusting the temporal phase of camera shake detection and correction. The digital image signal output from the analog signal processing unit 130 is stored in the image memory 151 through the memory controller 150, and is read out from the image memory 151 after delaying the time required for motion vector detection. The camera shake correction is performed by the memory controller 150 in accordance with the shake correction amount instructed by the system controller 170.

The digital image signal after image stabilization from the memory controller 150 is supplied to the filter 160. The filter 160 has a configuration of a circulating filter composed of digital circuits, and has a memory of one field or one frame. The output image signal is obtained from the filter 160. This output image signal is an image signal with an improved S / N ratio and image stabilization. Although not shown, the output image signal is compressed and recorded on a recording medium such as a memory card and displayed on an image display unit such as an LCD (Liquid Crystal Display).

The system controller 170 controls the driver 111, the timing generator 121, and the analog signal processing unit 130. In addition, the motion vector from the motion vector detector 141, the feature information from the feature extractor 142, and the luminance level from the brightness detector 180 are supplied to the system controller 170. In addition, the system controller 170 controls image stabilization by the memory controller 150 and adjusts the reliability of the motion vector based on the feature amount from the feature extractor 142 and the brightness level from the brightness detector 180. It determines and controls the filter coefficient of the filter 160 based on reliability.

In addition, the luminance detector 180 generates an AF (Auto Focus) signal for auto focus control, an auto exposure signal, and an auto white balance signal. These signals are supplied to the system controller 170, and a signal for controlling the imaging optical system 110 is formed by the system controller 170, and the formed control signal is supplied to the driver 111.

In addition, in order to reduce the shaking of the subject projected on the image pickup device 120 due to camera shake, the system controller 170 controls the timing generator 121 so that the electronic shutter speed is such that exposure blurring does not occur. In general, the shutter speed at which the exposure blur does not occur is said to be "1 / focal length (35 mm equivalent)". As the focal length, a value calculated by the system controller 170 for focus control is used.

In low light, the system controller 170 is a shutter speed such that the above-described exposure blurring does not occur instead of prolonged exposure (i.e., a slow shutter speed), and a predetermined interval, for example, an interval of one field or one frame. The image acceptance operation is controlled so that a plurality of images accept an image. The number of images to be accepted is the number corresponding to the brightness of the subject or a predetermined number of presets. In the case of a still camera, a plurality of images are the same image if the subject does not change and there is no camera shake. An image resulting from image stabilization and filter processing of the plurality of images received is an output image.

The motion detection unit 140 detects the overall motion of the screen by the representative point matching method, which is one of the motion detection methods by block matching. This method is not suitable for subjects that differ greatly between the frames because the subjects are assumed to be almost identical between the frames to be compared.

2 illustrates a configuration of an example of the camera shake detection unit 140. The image input 201 is an input portion of image data to be detected as a motion vector. Image data input from the image input 201 is supplied to a filter processing circuit 210 for removing frequency components unnecessary for motion detection. The output of the filter processing circuit 210 is supplied to the representative point extraction circuit 220. The representative point extraction circuit 220 extracts pixel data (referred to as representative points) at a predetermined position for each region composed of a plurality of pixels of the input image data, and stores the luminance level.

The representative point from the representative point extraction circuit 220 and the pixel data of the output of the filter processing circuit 210 are subtracted by the subtractor 230. The subtraction process is performed for each area. The absolute value conversion circuit 240 absoluteizes the difference signal output from the subtractor 230.

The motion vector detection circuit 250 detects the motion vector using the absolute value of the difference signal (hereinafter referred to as a residual difference as appropriate). The detected motion vector 260 is output. The filter processing circuit 210, the representative point extraction circuit 220, the subtractor 230, the absolute value conversion circuit 240, and the motion vector detection circuit 250 constitute the motion vector detection unit 141.

In addition, the evaluation value of the coordinate position indicated by the motion vector 260 is supplied to the feature extraction circuit 142. The feature extractor 142 outputs an evaluation value corresponding to the detected motion vector 260 as the feature information 261.

3 illustrates a method of detecting a motion vector using a representative point matching method. One picked-up image, for example, an image of one frame is divided into a plurality of areas. In FIG. 3, the detection area 301 is a search area for detecting a motion vector of a frame at time n, for example. In the example of the figure, the area | region of the size of (5x5) pixels is set. In this detection area 301, a pixel having the highest luminance correlation with the representative point is detected as a motion vector. The representative point 302 is set in the reference region 306 at the same position spatially as the detection region 301 of the frame at the time m. The representative point 302 exists only one pixel in the image of the time m used as a comparison source. The interval between the times n and m is an interval in which a plurality of images are received by continuous shooting, for example, one field or one frame.

The pixel 303 represents any one pixel in the detection area 301, and each pixel in the detection area 301 becomes a pixel to be compared with the representative point 302. The motion vector 304 shows an example of the detected motion vector. The pixel 305 shown with the diagonal line exists in the coordinate which a motion vector shows.

A plurality of pairs of the reference area 306 and the detection area 301 are set in each frame, and in each of them, the residual between the representative point 302 and the luminance level of the pixel in the detection area 301 is calculated. And the residual value in the position of each pixel of (5x5) of the detection area 301 is integrated, and the evaluation value of (5x5) regarding the one image 5x5 is calculated, and the evaluation value of In the distribution, the coordinates of the minimum position are detected as the motion vector.

The luminance level of the representative point of the coordinates (u, v) at the time m is expressed by km (u, v), and the luminance level at the coordinates (x, y) at the time n is kn. When expressed as (x, y), the residual expression in motion vector detection in the representative point matching method is expressed by the following equation (1).

[Equation 1]

The obtained residual relates to one pair of the reference region 306 and the detection region 301. Regarding a plurality of pairs existing throughout the frame, the residuals are similarly obtained, and the residuals are integrated at each of the coordinates (x, y), whereby an evaluation value of the coordinates (x, y) is obtained. In the example of Fig. 3, evaluation values are generated at (5 x 5) = 25 pixel positions, respectively.

An example of the relationship between the shift and the evaluation value is shown in FIG. 4. The difference between the coordinate (a) of the point where the evaluation value is the minimum and the minimum point and the representative point becomes the motion vector (mv (x, y)). When the representative point of the whole screen of one frame moves similarly to the position of the coordinate a, the evaluation value of the coordinate a becomes minimal. When expressed by a formula, it is represented by following formula (2). However, P (x, y) means the evaluation value (namely, the integrated value of the absolute value of a residual) in coordinate (x, y).

[Formula 2]

If the coordinate of the representative point is (0, 0), the motion vector can be expressed by the following equation (3).

[Equation 3]

The feature extraction unit 142 is a circuit that outputs the evaluation value at point a in FIG. 4. That is, the following formula (4) is output as the feature information (L).

[Equation 4]

The determination of the reliability of the detected motion vector will be described. FIG. 5 is a two-dimensional representation of the relationship between the deviation and the evaluation value shown in FIG. 4. For example, the change of the evaluation value at the time of cut | disconnecting in the plane parallel to one of an x-axis and a y-axis through the coordinate (a) which an evaluation value becomes minimum is shown. The minimum deviation is the coordinate on the x-axis or the coordinate on the y-axis of the point (a).

The solid line 401 in FIG. 5 shows the change of the evaluation value at the time of normal shaking. The absolute value of the evaluation value at the minimum point becomes sufficiently small at the time that a motion vector is detected, and the correlation between images is low at the time other than the minimum point. In this case, it can be determined that the reliability of the detected motion vector is high.

On the other hand, it is determined that the reliability of the motion vector detected based on the change in the evaluation value at the low contrast is low. The broken line 402 in FIG. 5 shows the change of the evaluation value at the low contrast. The absolute value of the evaluation value at the minimum point is sufficiently small at the shifted motion vector is detected, and the correlation between images is high at all shifts. However, the motion vector detected here is low in reliability because it is susceptible to noise and the like. In the case of low contrast, since the difference of the luminance level of the representative point and the luminance level of the pixel of a detection area becomes small as a whole, the evaluation value becomes small as a whole. As a result, the detection accuracy may be lowered under the influence of noise or the like, or motion vectors other than hand shake may be detected. Therefore, in order to increase the reliability of the detected motion vector, it is necessary to detect and exclude the case of low contrast.

In this one embodiment, it is determined whether or not it is low-contrast using the sum total of the evaluation values shown in Formula (5). That is, X and Y are the number of horizontal and vertical pixels of the detection area, and the sum of the evaluation values at each coordinate of the evaluation value P (x, y) is normalized to the number of pixels (ie, area) (X × Y). If S is obtained and the value S is small, it is determined that the subject is low contrast.

[Equation 5]

A process of obtaining the reliability Rs from the value S obtained in equation (5) will be described with reference to FIG. 8A. When the reliability is low, that is, when S is less than the threshold thra, the reliability Rs is set to zero. When the reliability is high, that is, when S is larger than the threshold thrB, the reliability Rs is set to one. In a case other than the above, shown in Fig. 8A, the following equation (6) is set.

[Equation 6]

The dashed-dotted line 403 in FIG. 5 shows the change in the evaluation value when the moving subject enters within the period in which the plurality of images are captured. On the side where the motion vector is detected, the absolute value of the evaluation value at the minimum point is relatively large, and the correlation between images is low on all the sides. When the moving subject enters, the correlation between the images is lowered, so the absolute value level of the evaluation value at the minimum is increased. Since the correlation between images is low, the reliability of the detected motion vector is low and cannot be used for correction. Therefore, reliability is determined using the evaluation value L in the minimum point. The evaluation value L in the minimum point can be expressed by equation (7). If L is large, it is determined that the moving subject is a mixed subject.

[Formula 7]

The process of obtaining the reliability R _L from the value L obtained in equation (7) will be described with reference to FIG. 8B. When the reliability is high, that is, when L is below the threshold thrc, the reliability R _L is set to one. When the reliability is low, that is, when L is larger than the threshold thrD, the reliability R _L is set to zero. In other cases, it is shown in Fig. 8B and set by the following equation (8).

Equation 8

The motion vector detector 141 obtains the reliability R obtained by integrating the two reliability Rs and R _L described above according to Equation (9), and outputs it to the system controller 170 together with the detected motion vector. . When the reliability R is small, the reliability of the motion vector is low. On the contrary, when the reliability R is large, the reliability of the motion vector is high. In addition, the evaluation value may be supplied from the motion vector detector 141 to the system controller 170, and the system controller 170 may calculate the reliability.

[Equation 9]

FIG. 6 shows a configuration of an example of the filter 160 in FIG. 1. The data X (z) 501 output from the memory controller 150 is input to the filter 160. The output Y (z) 502 is taken out from the adder 520 of the filter 160. Input data X (z) 501 is amplified k times in amplifier 510 and supplied to adder 520. The filter coefficient k is instructed by the system controller 170. However, 0≤k≤1.

The output data of the adder 520 is taken out as the output Y (z) and supplied to the delayer 530. The output data of the delayer 530 is supplied to the adder 520 through the amplifier 511. The amplifier 511 amplifies the signal of the delay unit 530 by (1-k) times. The delay unit 530 is a delay unit for delaying the output Y (z) 502 by one sampling period. One sampling period is a time difference between the reference region in which the representative point exists and the detection region, for example, one field or one frame.

In the filter 160 shown in FIG. 6, when k = 1, the output component of the time before it is supplied from the amplifier 511 to the adder 520 becomes 0, and input data X (z) 501 is returned. It is taken out as output data Y (z) 502 as it is. In the case of k ≠ 1, the output component at the time before it is supplied from the amplifier 511 to the adder 520 does not become 0, and in the adder 520, the output component X (z) 501 is used. The output component of the time is added. Since the signal components are correlated even at different times, since the random noise components do not have correlation, the noise components can be reduced by the addition processing in the adder 520.

Here, the filter coefficient Ky will be described. The filter coefficient Ky is calculated according to FIG. 8C by the signal level Y of the imaging device 120 outputted from the brightness detector 180. In the case of the luminance level in which a lot of noise components are included, that is, the filter coefficient Ky when Y is less than thrE can be expressed by equation (10). When the noise component has a low luminance level, that is, when Y is equal to or greater than thrE, the filter coefficient Ky is set to the predetermined filter coefficient Kmax.

Equation 10

The filter coefficient k in the filter 160 shown in FIG. 6 is calculated by the formula (11) from the filter coefficient Ky corresponding to the reliability R and the luminance level. However, only the initial image sets the filter coefficient k to 1 to eliminate the transient response of the filter.

[Equation 11]

A control operation performed by the system controller 170 will be described with reference to FIG. 7. This control operation is executed every interval, for example, one field or one frame, in which a plurality of images are captured, and the value of the counter indicates the number of executions. In low light, continuous shooting is performed at a shutter speed at which no exposure blur occurs, and a predetermined number of images obtained by continuous shooting are subjected to image stabilization and filter processing. When a predetermined number of images are processed and the value of the counter reaches a predetermined value, the processing is completed, and the resultant image is treated as the final captured image.

In step S10, each time a control operation is made, it is determined whether or not the value of the counter to ink ink (referred to as a counter) is zero. If it is correct (the counter is zero), the processing moves to step S11, and if it is wrong (the counter is not zero), the processing moves to step S20.

In step S11, the camera shake correction is set to OFF. That is, the memory controller 150 outputs the image as it is without performing image stabilization on the image stored in the image memory 151. In the case where the reliability is low, continuous processing is always performed at the beginning of the counter = 0 to prevent the camera shake correction signal from being output.

In step S12, the filter coefficient k of the filter 160 is set to one. Initial time is control to eliminate the transient response of the filter. In step S13, 1 is added to the counter. Then, the process ends. Processing of the next image inputted after one field or one frame starts from step S10.

In step S20, camera shake is detected and corrected. That is, the system controller 170 instructs the memory controller 150 in accordance with the motion vector detected by the motion vector detector 141 to instruct the memory controller 150 to remove the camera shake component from the image memory 151. Outputs image stabilization.

In step S21, as shown by the above formula (11), the coefficient k of the filter is calculated according to the reliability R and the filter coefficient Ky. In step S13, 1 is added to the counter and the processing ends. The filter coefficient k calculated by the system controller 170 is supplied to the filter 160, and the filter coefficient of the filter 160 is set to an appropriate value.

In one embodiment of the present invention described above, a motion vector is generated for the entire screen, but the screen may be divided into a plurality of blocks, and the camera shake correction and filter processing may be performed for each block. According to another embodiment of the present invention, the processing is performed for each block.

In other embodiment of this invention, the structure of the whole imaging device is the same as that shown in FIG. 1 which concerns on one embodiment mentioned above, and the code | symbol of the component of FIG. 1 is used also in description of other embodiment. In another embodiment, the camera shake detection unit 140 (including the motion vector detection unit 141 and the feature extraction unit 141) is configured to vertically hold the image pickup screen of the imaging device 120 in the vertical direction as shown in FIG. 9. Processing is performed for each (I x J) blocks formed by dividing and I-dividing in the lateral direction. A plurality of reference areas and detection areas are set in each block. Therefore, the camera shake correction operation by the memory controller 150 and the processing of the filter 160 are also performed for each block. The luminance detector 180 detects the luminance level of the signal output from the analog signal processor 130 for each block.

In low light, continuous shooting is performed at a shutter speed at which no exposure blur occurs, and the obtained multiple images are subjected to image stabilization and cyclic filter processing in the same manner as in the above-described embodiment. However, image stabilization and filter processing are performed for each block.

The motion vector detection by the representative point method is the same as that in the above-described embodiment (see FIG. 2), and as described with reference to FIG. 3, the luminance level and the detection of the representative point in the reference area are described. The absolute value of the residual with the luminance level of each pixel in the area 301 is calculated. In this case, the coordinates at the blocks i and j at time m (where i = 0, 1, 2, ..., I-1, j = 0, 1, 2, ..., J-1) If the luminance level at the representative point of (u, v) is kmi, j (u, v), and the luminance level at coordinates (x, y) at time n is kni, j (x, y) The residual expression in the motion vector detection in the representative point matching method is expressed by the following expression (12).

Equation 12

The obtained residual relates to one pair of the reference region 306 and the detection region 301. Similarly, for a plurality of pairs existing in the entire block, the residuals are obtained, and the residuals are accumulated in each of the coordinates (x, y), whereby an evaluation value of the coordinates (x, y) is obtained. In the example in one embodiment described above, evaluation values are generated at (5 x 5) = 25 pixel positions for each block.

The relationship between the deviation shown in FIG. 4 and an evaluation value is obtained for every block. The difference between the coordinates of the point a and the representative point where the evaluation value is the minimum and the minimum point becomes the motion vector mvi, j (x, y) of the block (i, j). When the representative point of the whole screen of one block moves similarly to the position of the coordinate a, the evaluation value of the coordinate a becomes minimal. Expressed by the equation, the motion vectors mvi, j (x, y) of each block are expressed by the following equation (13). However, Pi, j (x, y) means an evaluation value (that is, an integrated value of the absolute value of the residual) at the coordinates (x, y) of the blocks (i, j).

Equation 13

If the coordinate of the representative point is (0, 0), the motion vector of the block (i, j) can be expressed by the following equation (14).

[Equation 14]

The feature extraction unit 142 is a circuit for outputting an evaluation value corresponding to the motion vector obtained in each block. That is, the following equation (15) is output as the feature information of the blocks (i, j).

Equation 15

Fig. 10 is a two-dimensional representation of the relationship between the deviation for each block and the evaluation value. Similar to FIG. 5 in the above-described embodiment, the solid line 601 shows the change in the evaluation value during normal hand shake. The absolute value of the evaluation value at the minimum point becomes sufficiently small at the time that a motion vector is detected, and the correlation between images is low at the time other than the minimum point. In this case, it can be determined that the reliability of the detected motion vector is high.

On the other hand, it is determined that the reliability of the motion vector detected based on the change in the evaluation value at the low contrast is low. The broken line 602 in FIG. 10 shows the change of the evaluation value at the low contrast. On the side where the motion vector is detected, the absolute value of the evaluation value at the minimum point becomes sufficiently small, and the correlation between the images is high on all the sides. However, the motion vector detected here is low in reliability because it is susceptible to noise and the like. In the case of low contrast, since the difference between the luminance level of the representative point and the luminance level of the pixel of the detection area becomes smaller as a whole, the evaluation value becomes smaller as a whole. As a result, the detection accuracy may be deteriorated under the influence of noise or the like, and motion vectors other than hand shake may be detected. Therefore, in order to increase the reliability of the detected motion vector, it is necessary to detect and exclude the case of low contrast.

In another embodiment, it is judged whether or not it is low contrast by using the sum (Si, j) of the evaluation values for each block represented by the formula (16). That is, X and Y are the number of horizontal and vertical pixels of the detection area, and the sum of the evaluation values at each coordinate of the evaluation value Pi, j (x, y) is the number of pixels (ie, area) (X × Y). Normalized value (Si, j) is obtained, and when Si, j is small, it is judged that it is a low contrast subject.

[Equation 16]

The reliability (Rsi, j) from the values (Si, j) of the blocks (i, j) obtained in equation (16) is, as in the embodiment, when the reliability (Rsi, j) is lower than the threshold (thrA). ) Is set to zero. When the reliability is high, that is, when Si, j is larger than the threshold thrB, the reliability Rsi, j is set to one. In a case other than the above, it is set by following formula (17).

[Equation 17]

The motion vector output from the motion vector detection unit 141 in each of the divided blocks may include components caused by the moving subject in addition to the camera shake component. In order to reduce the malfunction caused by the moving subject, only the camera shake component needs to be extracted and the camera controller corrects the camera shake in accordance with the motion vector. In addition to the image stabilization, it is necessary to detect components other than the image stabilization in the detected motion vector and calculate the reliability R _L i, j so as not to be integrated in the filter 160.

The camera shake component can be expressed by Equation (18). However, MD represents a median filter. In the memory controller 150, this jitter component mvMD is instructed from the system controller 170 and corrected.

Equation 18

Detection and reliability of a moving subject will be described. The degree to which the motion subject exists in the corresponding block is determined by the degree of deviation between the motion vector component and the camera shake component. This deviation degree (Li, j) can be expressed by equation (19). When Li and j are large, it is determined that the moving subject is a mixed subject.

Formula 19

The reliability R _L i, j of the blocks i and j at this time is calculated by Li, j according to FIG. 8B as in the embodiment. When the reliability is high, that is, when Li, j is less than thrC, the reliability R _L i, j is set to one. When the reliability is low, that is, when Li, j is larger than thrD, the reliability R _L i, j is set to zero. In a case other than the above, equation (20) is set.

Equation 20

When the reliability of the motion vector detected by the motion vector detection unit 141 in each block i, j is Ri, j, the reliability Ri can be expressed by equation (21). When the reliability Ri (j) is small, the reliability of the motion vector is low. On the contrary, when the reliability Ri (j) is large, the reliability of the motion vector is high.

[Equation 21]

The filter to which the image of each block of which image stabilization is corrected is supplied has a configuration shown in FIG. 6 in one embodiment. However, filter coefficients are set for each block. The filter coefficient for each block is ki, j. The filter coefficient is (0 ≦ ki, j ≦ 1).

Here, the filter coefficients Kyi, j of the blocks i, j will be described. The filter coefficients Kyi, j are calculated in accordance with FIG. 8C by the signal levels Yi, j of the image pickup device 120 output from the luminance detector 180 as in one embodiment. In the case where the luminance level includes a large amount of noise components, that is, when Yi, j is less than thrE, the filter coefficients Kyi, j can be expressed by equation (22). When the noise component has a low luminance level, that is, when Yi, j is equal to or greater than thrE, the filter coefficients Kyi, j are set to predetermined filter coefficients Kmax.

Formula 22

Similarly to one embodiment, the filter coefficients ki and j are calculated by equation (23) by the filter coefficients Kyi, j corresponding to the reliability R and the luminance level. However, for the initial image only, the filter coefficient k is set to 1 to eliminate the transient response of the filter.

Formula 23

The control operation performed by the system controller 170 is the same as that of FIG. 7 described in one embodiment. In one embodiment, however, the control operation is performed for one field or one frame, but it is necessary to operate for each block. Then, processing is performed on all blocks of the predetermined number of images, and when the value of the counter reaches a predetermined value, the processing is completed, and the resulting image is treated as the final captured image.

Since the above-described other embodiments of the present invention individually control each of the plurality of blocks as compared with the one embodiment, the filter coefficients can be controlled in accordance with the local characteristics of the image, and the image quality can be further improved. have.

As mentioned above, although embodiment of this invention was described concretely, this invention is not limited to embodiment mentioned above, Various deformation | transformation based on the technical idea of this invention is possible. For example, motion vector detection is not limited to the representative point method, but other methods may be used.

According to the present invention, a plurality of images are acquired by performing continuous shooting at an accumulation time at which exposure blur does not occur even in a low illuminance state, thereby producing an image free of image blur by correcting the image blur between the plurality of images, and further rotating By filtering with a type filter, an image having an improved S / N ratio can be generated. By using the cyclic filter in the time direction, unlike the simple addition method, it can be accepted over a long period without limit to the number of images. According to the present invention, the image quality can be improved by controlling the filter coefficients according to the reliability in consideration of the decrease in the reliability of the motion vector due to the low contrast and the unnecessary moving subject during image capturing.

Claims (8)

An image capturing step of capturing a plurality of images different in time, A motion detection step of detecting motion from the plurality of images and generating motion information; A reliability determination step of determining the reliability of the motion information; A camera shake correction step of correcting camera shake occurring between the plurality of images in accordance with the motion information; A filter step for filtering the image after image stabilization by a cyclic filter, And the characteristic of the filter process is changed in accordance with the reliability of the motion information. The method of claim 1, In the image capturing step, each of the plurality of images is captured as an accumulation time such that exposure blur does not occur. The method of claim 1, The image stabilization method is not performed with respect to an image of a head of the plurality of images. The method of claim 1, The characteristic of the said cyclic filter is changed about the image of the head of the said multiple image. The imaging method characterized by the above-mentioned. The method of claim 1, And in the reliability determination step, the motion information obtained from an image photographing a low-contrast subject is determined that the reliability is low. The method of claim 1, And in the reliability determination step, the motion information obtained from an image in which a moving subject exists in a part of the plurality of images determines that the reliability is low. The method of claim 1, An image pickup method characterized by dividing a captured image into a plurality of blocks, and changing the characteristics of the motion detection step, the reliability determination step, the camera shake correction step, the filter step, and the filter process for each block. . Image capturing means for capturing a plurality of images different in time; Motion detection means for detecting motion from the plurality of images and generating motion information; Reliability determination means for determining the reliability of the motion information; Hand shake correction means for correcting hand shake occurring between the plurality of images in accordance with the motion information; Filter means for filtering the image after image stabilization by a cyclic filter, And a characteristic of the filter means in accordance with the reliability of the motion information.

Images