JP5377073B2 - Imaging apparatus, blur repair method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to inexpensively obtain an opening pattern of a nonperiodic shutter with certainty. <P>SOLUTION: A plurality of nonperiodic opening patterns is held as a list by opening pattern holding means. By selecting one opening pattern among the plurality of nonperiodic opening patterns, the opening pattern of a shutter, which is used for photographing, is decided. A blur restoration circuit 428 irregularly opens and closes the shutter during the specified exposure time period, and carries out an encoding opening process of restoring the blur using the correlation of an obtained image and the opening pattern of the shutter. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a technique for blurring restoration of digital image data.

  2. Description of the Related Art In recent years, attention has been paid to camera shake correction processing at the time of shooting as the resolution of an image sensor of a digital camera increases. Particularly, since an image can be easily enlarged and confirmed to 100% or more on a personal computer, how to correct camera shake is an important theme.

  As a camera shake correction method, there are a hardware implementation method and a software implementation method. As a realization method by hardware, a method of reducing camera shake by mounting a gyro sensor on a camera and driving a lens or an image sensor so as to cancel vibration obtained from the gyro sensor during exposure can be considered. For example, Patent Document 1 discloses an implementation method by correcting a lens optical system.

  However, the hardware implementation method has a problem that the number of parts and the manufacturing cost increase. Therefore, in order to realize the same function with an inexpensive digital camera, it can be realized by software.

  As a realization method by software, a method of preparing and synthesizing a plurality of images exposed for a short time can be considered. For example, one disclosed in Patent Document 2 is one of them. In Patent Document 2, camera shake correction is realized by correcting and synthesizing the shakes of a plurality of images that are continuously photographed with an exposure time that allows for shake.

  On the other hand, in recent years, a technique has also been developed in which an aperture pattern of a shutter is coded and camera shake is corrected by using the coded aperture pattern. For example, an image is taken with a randomly coded exposure time, and blurring is corrected by applying an inverse matrix of the exposure time code.

JP 2002-214657 A JP 2007-243774 A

  However, the method of randomly encoding the shutter opening pattern as described in Non-Patent Document 1 has the following problems. Conventionally, in order to generate this random opening pattern, a method using a pseudo-random number is generally used. However, such a method does not guarantee that a pattern without periodicity is generated reliably. If periodicity appears in this pattern, there is a problem that the image quality of the blur-corrected image is deteriorated. In addition, when mounting on a digital camera or the like is considered, there is a problem that the calculation cost of pseudo random numbers is increased.

  The present invention has been made in view of the above situation, and an object of the present invention is to make it possible to reliably obtain a shutter opening pattern having no periodicity at a low cost.

An image pickup apparatus according to the present invention is an image pickup apparatus including a shake detection unit, and includes an opening pattern holding unit that holds an opening pattern of a shutter having no periodicity, and an opening pattern from the opening pattern held by the opening pattern holding unit. An aperture pattern determining means for determining the image, an imaging means for performing imaging using the aperture pattern determined by the aperture pattern determining means, blur information detected by the blur detecting means, and a point image obtained from the aperture pattern A blur correction corrector that obtains a blur-repaired image by applying a function inverse filter to the captured image; and a shooting condition acquisition unit that acquires a shooting mode that is a shooting condition; and the aperture pattern determination unit includes the shooting pattern mode determines the opening pattern having closest frequency characteristic pink noise characteristics during portrait mode And wherein the door.

  According to the present invention, it is possible to acquire a shutter opening pattern having no periodicity at a low cost and perform photographing, and it is possible to suppress deterioration in image quality when performing blur correction.

It is a perspective view which shows the external appearance of a digital camera. It is a vertical sectional view showing the internal structure of the digital camera. It is a block diagram which shows the circuit structure of a digital camera. It is a flowchart which shows the flow of the blurring correction process by an encoding opening process. It is a figure for demonstrating the concept of the principle of a blurring correction process. It is a flowchart which shows the production method of PSF. It is a figure which shows the example of PSF in a general rotational motion. It is a figure which shows the example and frequency characteristic of PSF in the case of opening. It is a figure which shows the example and frequency characteristic of PSF at the time of controlling the opening / closing conditions of a shutter. It is a figure which shows the image which raise | generated rotation blur, and the image which correct | amended rotation blur. It is a figure which shows the list | wrist of the opening pattern in 1st Embodiment. It is a figure for demonstrating the determination method of the opening pattern in 2nd Embodiment. It is a flowchart which shows the flow of the determination process of the opening pattern in 3rd Embodiment. It is a figure for demonstrating the selection method of the candidate of an opening pattern. It is a flowchart which shows the flow of the determination process of the opening pattern in 4th Embodiment. It is a flowchart which shows the flow of the determination process of the opening pattern in 5th Embodiment.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
(First embodiment)
FIG. 1 is a perspective view showing an external appearance of a digital camera that is an imaging apparatus according to the present embodiment, FIG. 2 is a vertical sectional view of FIG. 1 showing an internal structure of the digital camera, and FIG. 3 is a block diagram showing a circuit configuration of the digital camera. It is.

  As shown in FIG. 1, an eyepiece window 111 for finder observation, an AE (automatic exposure) lock button 112, an AF (autofocus) distance measuring point selection button 113, and a shooting operation are provided on the upper part of the camera body 100. The release button 114 is provided. An electronic dial 411, a shooting mode selection dial 117, and an external display device 409 are provided. The electronic dial 411 is a multi-function signal input device that is used in combination with other operation buttons to input numerical values to the camera and to switch the shooting mode. The external display device 409 includes a liquid crystal display device, and displays shooting conditions such as shutter speed, aperture, and shooting mode, and other information.

  On the back of the camera body 100, there are provided an LCD monitor device 417 for displaying a photographed image and various setting screens, a monitor switch 121 for turning on / off the LCD monitor device 417, a cross switch 116, and a menu button 124. It has been. The cross switch 116 has four buttons arranged vertically and horizontally, and a SET button arranged in the center. The user instructs the camera to select and execute menu items displayed on the LCD monitor device 417. Used to do. The menu button 124 is a button for causing the LCD monitor device 417 to display a menu screen for performing various camera settings. For example, when selecting and setting the shooting mode, after pressing the menu button 124, the user operates the up / down / left / right buttons of the cross switch 116 to select the desired mode, and the desired mode is selected. Setting is completed by pressing the SET button. The menu button 124 and the cross placement switch 116 are also used for setting an AF mode, which will be described later.

  In FIG. 2, reference numeral 100 denotes a camera body, 200 denotes a photographing lens of an imaging optical system, and the photographing lens 200 is detachable from the camera body 100 via a lens mount 202. Reference numeral 201 denotes a photographing optical axis.

  Reference numeral 203 denotes a quick return mirror disposed in the photographing optical path, which retracts the subject light from the photographing lens 200 to the viewfinder optical system (the position shown in FIG. 2, the oblique position) and out of the photographing optical path. It can move between positions (referred to as retreat positions).

  Reference numeral 204 denotes a focusing plate, on which the subject light guided from the quick return mirror 203 to the finder optical system is imaged. Reference numeral 205 denotes a condenser lens for improving the visibility of the finder, and 206 denotes a pentagonal roof prism.

  Reference numerals 209 and 210 respectively denote a rear curtain and a front curtain that constitute a shutter. When the rear curtain 209 and the front curtain 210 are opened, an image sensor 418 that is a solid-state image sensor disposed behind is exposed for a necessary time. At this time, according to the shutter opening pattern determined as will be described later, the shutter is opened and closed irregularly during the designated exposure time, and photographing is performed. The image data is processed by the A / D converter 423, the image processing circuit 425 (see FIG. 3), etc. in a state where time-division exposure is performed at a predetermined time t. The image processing circuit 425 adds and synthesizes the image data selected according to the output of the exposure condition generation circuit 427 and outputs it. The output composite image data is output to the external storage device 419 and the like by applying a blur correction (blur correction) process using the exposure condition and motion information in the blur correction circuit 428. Details of the blur repair process will be described later.

  Reference numeral 211 denotes a printed circuit board, and an image sensor 418 is held on the printed circuit board 211. A display board 215 that is another printed board is disposed behind the printed board 211. An LCD monitor device 417 and a backlight illumination device 416 are disposed on the opposite surface of the display substrate 215.

  Reference numeral 419 denotes an external storage device that records image data, 217 denotes a battery (portable power supply), and the external storage device 419 and the battery 217 are detachable from the camera body 100.

  In FIG. 3, the microcomputer 402 controls the operation of the entire camera, including processing of image data output from the image sensor (CCD in this embodiment) 418 and display control of the LCD monitor device 417.

  The switch (SW1) 405 is turned on when the release button 114 (see FIG. 1) is half-pressed, and when the switch (SW1) 405 is turned on, the digital camera of this embodiment is in a shooting preparation state. The switch (SW2) 406 is turned on when the release button 114 is pressed to the end (fully pressed state). When the switch (SW2) 406 is turned on, the digital camera according to the present embodiment starts a photographing operation.

  The lens control circuit 407 performs communication with the photographic lens 200, drive control of the photographic lens 200 during AF, and drive control of the diaphragm blades. An external display control circuit 408 controls an external display device (OLC) 409 and a display device (not shown) in the finder. The switch sense circuit 410 transmits signals of a large number of switches including the electronic dial 411 provided in the camera to the microcomputer 402. The strobe light emission dimming control circuit 412 is grounded via the X contact 412a and controls the external strobe.

  A distance measuring (AF) circuit 413 detects a defocus amount for a subject for AF. A photometry (AE) circuit 414 measures the luminance of the subject. A shutter control circuit 415 controls the shutter and performs appropriate exposure on the image sensor 418.

  The LCD monitor device 417 of the present embodiment is a transmissive type, and an image cannot be visually recognized only by driving the LCD monitor device, and a backlight illumination device 416 is necessarily provided on the back surface thereof as shown in FIG. . As described above, the LCD monitor device 417 and the backlight illumination device 416 constitute an image display device.

  The external storage device 419 is, for example, a hard disk drive or a semiconductor memory card that can be attached to and detached from the camera body 100.

  Further, the microcomputer 402 is connected to an A / D converter 423, an image buffer memory 424, an image processing circuit 425 including a DSP, a motion information detection circuit 426 as a blur detection means, an exposure condition generation circuit 427, and a blur repair circuit 428. Has been. When the switch (SW2) 406 is turned on, the image processing circuit 425 applies sharpness processing to the image data using a distance map obtained from a distance map generation circuit (not shown).

  The blur repair circuit 428 implements blur repair using a technique called coded exposure. The coded aperture is a technology for irregularly opening and closing the shutter during a specified exposure time and restoring the blur using the correlation between the obtained image and the aperture pattern of the shutter. Hereinafter, the basic flow of the coded aperture processing will be described by taking as an example the case of correcting blur in the rotation direction of an image. After that, a method for realizing the coded aperture processing in this embodiment will be described. Note that the blur repair circuit 428 functions as a blur repair correct means for obtaining a blur-corrected image by applying an inverse filter of a point spread function obtained from the blur information and the aperture pattern to the photographed image.

<Outline of coded aperture processing>
First, the flow of blur correction processing by the encoded aperture processing will be described using the flowchart of FIG. For example, in photographing with a normal camera, input photographing data is considered (step S401). At this time, assuming that the intensity of light incident per unit time at (x, y) is I (x, y), the angular velocity is ω (t), and the exposure time is T, the camera has I (x, y at time t. Since information obtained by rotating y) by −θ (T−t) is incident, the photographing data I_blur (x, y) is expressed by the following equations (1) to (3).

  The origin of the coordinate system is matched with the input rotation center coordinates. In equation (1), h (t) is a function representing shutter opening / closing information (opening pattern), and takes a value of 1 or 0. 1 indicates that the shutter is open, and 0 indicates that the shutter is closed. In the transformation from Equation (1) to Equation (2), the integration variable is converted. ω (t) = dθ / dt, and ω (θ) is a function in which ω (t) is rewritten as a variable of θ using the relationship between t and θ. Similarly, h (θ) is a function obtained by rewriting h (t) to θ using the relationship between t and θ. Here, φ = θ (T). In Expression (3), h ′ (θ) = h (θ) / ω (θ).

  Next, exposure conditions are acquired (step S402). The exposure condition in this case is the shutter opening pattern during the exposure time.

  Next, the relationship between the blur angle and time is obtained (step S403).

  Next, PSF h ′ (θ) in polar coordinates is calculated from the blur angle and exposure conditions (step S404). Here, PSF is an abbreviation for Point Spread Function (point spread function, point spread function). A detailed calculation method of PSF h ′ (θ) will be described later.

  Next, the photographing data is converted into polar coordinates (step S405). As described above, the origin of the orthogonal coordinate system at this time is matched with the rotation center coordinate of the input image. When polar coordinate conversion is performed, Equation (3) becomes Equation (4) below.

  Here, (x, y) = r (cos Θ, sin Θ) in equation (3). This is the same formula as the shake for translation, and can be regarded as having been subjected to convolution with PSF h ′ (θ).

  Since this is a theoretical formula and the actual data is digital, some kind of interpolation is required to convert polar coordinates from real space. Any interpolation method may be used. Here, bilinear interpolation is used.

  Subsequently, deconvolution is performed so as to cancel the convolution of Expression (4) (step S406). Any existing algorithm may be used as the deconvolution algorithm at this time. For example, division in a frequency space, Lucy-Richardson algorithm, algorithm using a Wiener filter, algorithm using a regularization filter, and the like can be given. In this example, it is assumed that the shape of h ′ (θ) is controlled by controlling the opening pattern, and division in the frequency space is performed. Details of the deconvolution will be described later.

  Since I (r, Θ) is obtained by deconvolution, this I (r, Θ) is returned to the real space display I (x, y) (step S407). This conversion processing also requires interpolation processing as in the conversion from real space to polar coordinates.

  Thereafter, the data converted into the real space is output as the data after the blurring restoration (step S408).

  With the processing described above, it is possible to realize image blurring restoration processing for an image obtained by opening and closing a shutter that is irregularly changed from the opening pattern and blurring information. FIG. 5 conceptually shows the principle of the blur repair process. FIG. 5 shows a flow in which a rotational shake is converted into a horizontal shake by polar transformation, the horizontal shake is removed by deconvolution, and finally returned to the real space to restore the rotational shake. Yes.

  In this example, in order to deal with the blur correction process in the rotation direction, the process is applied by converting to polar coordinates. However, by performing the process without converting to the polar coordinates as described above, so-called horizontal and vertical directions are performed. Shift blur correction is also possible. For example, it is conceivable that after the blur in the horizontal and vertical directions is first repaired without conversion to polar coordinates, the above-described blur correction process in the rotational direction is executed.

<PSF creation method>
In the present embodiment, the motion information detection circuit 426 can obtain an angle-time relationship θ (t). A method for creating a PSF will be described with reference to the flowchart of FIG. First, the angular velocity ω (t) is calculated by differentiating the angle with time (step S601). By combining this with θ (t), the angular velocity can be expressed as a function of θ. Let this be ω (θ).

  Further, from the exposure condition and angle, the function h (t) representing the shutter opening / closing information (opening pattern) can be similarly represented by the function of θ. This is set to h (θ) (step S602).

  From these pieces of information, the following equation (5) is calculated (step S603), and h ′ (θ) is output as PSF (step S604). From equation (3), this is the PSF on polar coordinates.

  An example of PSF in a general rotational motion is shown in FIG. In FIG. 7, the horizontal axis represents an angle (unit: radians), and the vertical axis represents a PSF value. The opening condition at this time is assumed to be h (t) = 1 (0 ≦ t ≦ T) h (t) = 0 else. Further, it is assumed that the angular velocity is accelerating. As ω (θ) increases, the value of PSF h ′ (θ) decreases.

<Method of deconvolution>
When Expression (4) is converted into the frequency space, the following Expression (6) is obtained. In (f, ρ), f is a variable corresponding to the frequency conversion of r, and ρ is a variable corresponding to the frequency conversion of Θ.

  Since H ′ (f, ρ) is known, I (f, ρ) can be obtained in principle by dividing I_blur (f, ρ) by H ′ (f, ρ) in the frequency space. But there is a problem here. In the following, it is assumed that ω (θ) is a constant, considering blurring due to a constant angular velocity motion.

  First, consider the case of h (t) = 1 (0 ≦ t ≦ T) h (t) = 0 else, which is a normal exposure condition. Here, h (t) = 1 represents a state where the shutter is open, and h (t) = 0 represents a state where the shutter is closed. FIG. 8A shows the shape of the PSF at this time, and FIG. 8B shows the frequency converted PSF. In FIG. 8, the horizontal axis represents the angle (unit is radians), and the vertical axis represents the PSF value in FIG. 8A and the absolute value of H ′ (f, ρ) in FIG. 8B. From FIG. 8 (b), it can be seen that, under normal exposure conditions, points appear where the value of H ′ (f, ρ) periodically becomes zero. This means that information corresponding to the frequency has been lost.

  In such a state, an error (waveform pattern) occurs in a portion corresponding to the frequency at which information is lost on the deconvolved output image. A technique for preventing this error is a coded aperture. For example, it is possible to prevent the frequency characteristics of the PSF from becoming 0 by randomly changing the order and length of 0 and 1 representing opening and closing of the shutter. FIG. 9A shows the shape of the PSF at this time, and FIG. 9B shows the PSF frequency-converted. In FIG. 9, the vertical axis and the horizontal axis are the same as those in FIG. In this case, since no information is lost at any frequency, it is theoretically possible to perform deconvolution completely by dividing by H ′ (f, ρ).

  Lastly, FIG. 10 schematically shows a result of generating a blurring image in the rotation direction by simulation and applying a blurring repair process using a coded aperture. FIG. 10A shows the blurred image, and FIG. 10B shows the application result of the blur restoration process.

  As described above, a suitable blurring restoration result can be obtained by applying the blurring restoration process using the coded aperture. It can also be seen that the information necessary for this processing is a blurred image generated by adding and synthesizing according to the aperture pattern, motion information, and aperture pattern.

  Next, a method for determining an opening pattern, which is the main focus of the present invention, will be described. In the present embodiment, a plurality of opening patterns having no periodicity are held as a list by the opening pattern holding means, and one opening pattern is selected from them, thereby determining the opening pattern to be used for photographing.

  FIG. 11 is a diagram showing a list of opening patterns. An ID and an opening pattern are registered in the list, and each is in the form of a table associated one-to-one. Here, “1” of the opening pattern indicates that the shutter is open, and “0” indicates that the shutter is closed, and each opening pattern in the list is designed in advance so as not to have periodicity. ing.

  Here, the pattern having no periodicity means a pattern in which the frequency characteristic of the PSF does not become zero, as shown in FIG. In designing an opening pattern that does not have periodicity, for example, it is conceivable to use a pseudo-random number generation method such as Mersenne Twister, which is known to have a very long period. Then, after verifying that the generated random number has no periodicity, it is registered in the list. Of course, it is not always necessary to use a pseudo-random number for the design method of the opening pattern, and any method may be used as long as the finally generated opening pattern has no periodicity. Further, the verification that there is no periodicity may be performed by performing frequency conversion on the PSF and confirming that the absolute value of H ′ (f, ρ) does not fall to zero, as shown in FIG. .

  In this embodiment, at the time of shooting, the exposure condition generation circuit 427 determines an opening pattern to be used for shooting by, for example, randomly selecting one from the list of opening patterns. This is an example of processing by the opening pattern determination means in the present invention.

  The blur correction processing for the image data after photographing is performed by the blur repair circuit 428 according to the flow of FIG. 4 described above. Here, the aperture pattern selected from the aperture pattern list at the time of shooting is acquired as the exposure condition in step S402.

  With the above processing, it is possible to reliably select and use an aperture pattern that does not have periodicity at the time of shooting, and thus it is possible to prevent the frequency characteristics of the PSF from becoming zero. Therefore, when image blur correction is performed, it is possible to suppress image quality deterioration of the image blur correction image. In addition, since the opening pattern is determined by referring to the table, the calculation cost can be reduced as compared with the conventional method in which the pattern is dynamically determined by a pseudo random number or the like.

(Second Embodiment)
In the first embodiment, the method for determining the aperture pattern to be used for photographing by selecting one from a list of aperture patterns having no periodicity prepared in advance has been described. In the present embodiment, a second method for determining an opening pattern will be described.

  In the present embodiment, as shown in FIG. 12, an opening pattern having a sufficiently long periodicity is prepared in advance, and a part of the opening pattern is randomly extracted to determine the opening pattern used for photographing. As described above, the pattern generated by extracting a part of the non-periodic pattern itself does not necessarily have periodicity, so that it is possible to surely shoot using an opening pattern without periodicity. It becomes like this.

  As a method of generating an opening pattern without a sufficiently long periodicity, the method using pseudo random numbers described in the first embodiment can be considered. In the present embodiment as well, as in the first embodiment, it is not always necessary to use a pseudo-random number for generating the opening pattern, and any method can be used as long as the generated opening pattern has no periodicity. May be used. Of course, it is not necessary to have one long opening pattern prepared in advance. A plurality of opening patterns are prepared in the form of a list as shown in the first embodiment, and a part of them is extracted as an opening pattern. You may do it.

  The blur correction process for the image data after shooting is performed according to the flow of FIG. 4 described in the first embodiment. Here, as an exposure condition in step S402, an opening pattern determined by extracting a part of a previously prepared pattern without sufficiently long periodicity is acquired.

  With the above processing, an aperture pattern that does not have periodicity can be used reliably at the time of shooting, so that the frequency characteristics of the PSF can be prevented from becoming zero. Therefore, when image blur correction is performed, it is possible to suppress image quality deterioration of the image blur correction image. In addition, the calculation cost can be reduced as compared with a conventional method in which a pattern is dynamically determined by a pseudo random number or the like.

(Third embodiment)
In the first embodiment and the second embodiment, the method of randomly determining one opening pattern has been described. For example, a plurality of candidates are determined by these methods, and further, the candidates are determined from the candidates according to the shooting conditions. A final opening pattern may be determined. In the present embodiment, a method for determining a pattern when there are a plurality of candidate opening patterns will be described.

  In the present embodiment, one of a plurality of opening pattern candidates is determined according to the exposure time at the time of shooting. Details of this processing will be described with reference to the flowchart of FIG. In step S1301, a plurality of opening pattern candidates are selected. As shown in FIG. 14A, this may be performed by extracting a plurality of patterns at random from a sufficiently long pattern without periodicity. Alternatively, as shown in FIG. 14B, there may be a plurality of patterns having no periodicity, and a part of each pattern may be extracted at random. Further, each entry in the opening pattern list as described in the first embodiment may be regarded as a plurality of candidates.

  In step S1302, an average value of the patterns is calculated for each of the opening pattern candidates selected in step S1301. This is obtained by adding each bit of the pattern and dividing by the bit length. That is, this average value is 1 if it is fully open (all patterns are 1) and 0 if it is fully closed (all patterns are 0). Therefore, the average value calculated here falls between 0 and 1. In the example shown in FIG. 14, the average value of each candidate is shown in parentheses.

  In step S1303, the exposure time at the time of shooting is acquired.

  In step S1304, the exposure time acquired in step S1303 is compared with a predetermined exposure time threshold value. If the exposure time is smaller than the threshold value, the process proceeds to step S1305; otherwise, the process proceeds to step S1306.

  In step S1305, the average values of the aperture pattern candidates calculated in step S1302 are compared, the pattern having the maximum average value is determined as the aperture pattern used for imaging, and the process ends.

  On the other hand, in step S1306, the average values of the aperture pattern candidates calculated in step S1302 are compared, the pattern having the minimum average value is determined as the aperture pattern used for imaging, and the process ends.

  By the above processing, when there are a plurality of opening pattern candidates, an appropriate opening pattern can be determined. In addition, by controlling the aperture pattern according to the exposure conditions as described above, the overall exposure amount can be increased, and blur correction with reduced noise can be performed.

  In the present embodiment, the method of controlling the aperture pattern according to the exposure conditions has been described, but it goes without saying that the same effect can be obtained even when other imaging conditions are used. For example, the aperture pattern may be controlled using ISO sensitivity as a parameter. When the ISO sensitivity is used as a parameter, the same effect can be obtained by selecting a pattern with a large exposure, that is, a pattern with a large average value when the sensitivity is high.

(Fourth embodiment)
In this embodiment, as a pattern determination method when there are a plurality of opening pattern candidates, a process of determining one of a plurality of opening pattern candidates according to a shooting mode that is a shooting condition will be described.

  Generally, in a portrait photograph taken in the portrait mode, many flat portions appear in the photographed image. In other words, the image often has many low-frequency components. In such a case, if shooting and blur correction are performed using an opening pattern of a random noise pattern that has a response value in the entire frequency range, noise in the blur correction image is noticeable because it has unnecessary frequency components. Problem arises. In the present embodiment, as a method for solving this problem, a method for controlling the selection of the aperture pattern according to the photographing mode will be described.

  FIG. 15 is a flowchart showing a method for determining an opening pattern in the present embodiment. In step S1501, a plurality of opening pattern candidates are selected. As shown in FIG. 14A, this may be performed by extracting a plurality of patterns at random from a sufficiently long pattern without periodicity. Alternatively, as shown in FIG. 14B, there may be a plurality of patterns having no periodicity, and a part of each pattern may be extracted at random. Further, each entry in the opening pattern list as described in the first embodiment may be regarded as a plurality of candidates.

  In step S1502, the frequency of each of the opening pattern candidates selected in step S1501 is converted.

  In step S1503, the shooting mode is acquired.

  In step S1504, it is determined whether the shooting mode is a portrait mode. If it is the portrait mode, the process proceeds to step S1505; otherwise, the process proceeds to step S1506.

  In step S1505, a pattern having a frequency characteristic closest to the pink noise characteristic is determined as an opening pattern from the opening pattern candidates subjected to frequency conversion in step S1502, and the process ends.

  On the other hand, in step S1506, a pattern having a frequency characteristic closest to the random noise characteristic is determined as an opening pattern from the candidates for the opening pattern subjected to frequency conversion in step S1502, and the process ends.

  With the above processing, it is possible to reduce the occurrence of noise when blur correction is performed on the portrait image including a large amount of low frequency components by the encoded aperture processing.

(Fifth embodiment)
In this embodiment, as a pattern determination method when there are a plurality of aperture pattern candidates, a process of determining one of a plurality of aperture pattern candidates according to the f-value at the time of shooting, which is a shooting condition, will be described.

  When the f value at the time of shooting is small, the depth of field becomes shallow, so that there are more blurred portions in the image than when the f value is large. That is, the image often has more low frequency components. In such a case, if shooting and blur correction are performed using an opening pattern of a random noise pattern that has a response value in the entire frequency range, noise in the blur correction image is noticeable because it has unnecessary frequency components. Problem arises. In the present embodiment, as a method for solving this problem, a method for controlling the selection of the aperture pattern according to the f value at the time of shooting will be described.

  FIG. 16 is a flowchart showing a method for determining an opening pattern in the present embodiment. In step S1601, a plurality of opening pattern candidates are selected. As shown in FIG. 14A, this may be performed by extracting a plurality of patterns at random from a sufficiently long pattern without periodicity. Alternatively, as shown in FIG. 14B, there may be a plurality of patterns having no periodicity, and a part of each pattern may be extracted at random. Further, each entry in the opening pattern list as described in the first embodiment may be regarded as a plurality of candidates.

  In step S1602, frequency conversion is performed on each of the opening pattern candidates selected in step S1601.

  In step S1603, the set f value is acquired.

  In step S1604, the f value acquired in step S1603 is compared with a threshold value. If the f value is smaller than the threshold value, the process proceeds to step S1605; otherwise, the process proceeds to step S1606.

  In step S1605, a pattern having a frequency characteristic closest to the pink noise characteristic is determined as an opening pattern from the candidates for the opening pattern subjected to frequency conversion in step S1602, and the process ends.

  On the other hand, in step S1606, a pattern having a frequency characteristic closest to the random noise characteristic is determined as an opening pattern from the candidates for the opening pattern subjected to frequency conversion in step S1602, and the process ends.

  With the above processing, noise is generated when blurring correction is performed by the coded aperture processing on an image that contains a lot of blur, that is, an image that contains a lot of low-frequency components. Can be reduced.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions or flowcharts of the above-described embodiments to a system or apparatus. In this case, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

Claims (5)

An image pickup apparatus including a shake detection unit,
An opening pattern holding means for holding the opening pattern of the shutter having no periodicity;
An opening pattern determining means for determining an opening pattern from the opening pattern held by the opening pattern holding means;
Photographing means for photographing using the opening pattern determined by the opening pattern determining means;
Blur information detected by the blur detection unit, and a blur correction normal unit that obtains a blurred image by applying an inverse filter of a point spread function obtained from the aperture pattern to the captured image ,
A shooting condition acquisition means for acquiring a shooting mode that is a shooting condition;
The aperture pattern determining means determines an aperture pattern having a frequency characteristic closest to a pink noise characteristic when the shooting mode is a portrait mode . The opening pattern holding means holds a list of opening patterns;
The imaging apparatus according to claim 1, wherein the opening pattern determination unit determines an opening pattern by selecting an opening pattern from the list of opening patterns.   The imaging apparatus according to claim 1, wherein the pattern determining unit determines the opening pattern by extracting a part of the opening pattern held by the opening pattern holding unit. A blur repair method using an imaging device including a blur detection means,
Determining an opening pattern from the opening pattern of the non-periodic shutter previously held by the opening pattern holding means;
Photographing using the determined aperture pattern;
Possess and obtaining the blur information detected by the blur detecting means, the repaired image blur by applying an inverse filter to the captured image of the point spread function derived from the opening pattern,
In the step of determining the aperture pattern, a blur repair method using an imaging apparatus , wherein an aperture pattern having a frequency characteristic closest to a pink noise characteristic is determined when the photographing mode is a portrait mode . A program for controlling an imaging apparatus provided with a shake detection means,
An opening pattern holding means for holding the opening pattern of the shutter having no periodicity;
Pattern determining means for determining an opening pattern from the opening pattern held by the opening pattern holding means;
Photographing means for photographing using the opening pattern determined by the pattern determining means;
Blur information detected by the blur detection unit, and a blur correction normal unit that obtains a blurred image by applying an inverse filter of a point spread function obtained from the aperture pattern to the captured image ,
A photographing condition acquisition section that acquires photographing mode is an imaging condition to function the computer,
The opening pattern determining means is a program for determining an opening pattern having a frequency characteristic closest to a pink noise characteristic when the photographing mode is a portrait mode .

Images