JP6376769B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging device or the like that can detect defective pixels.

  In general, in an image pickup device that is mounted on an image pickup apparatus such as a digital still camera or a digital video camera and picks up a subject, a defect (scratch) may occur in some of a large number of pixels. Such a defect occurs due to various causes such as generation of dark current and abnormality of the photodiode.

  The defective pixel outputs an abnormal level signal. For example, when the output level of the defective pixel is higher than the original output level, it is called a white flaw, and when it is low, it is called a black flaw. When the captured image is displayed on the screen based on the output signal of the image sensor having such defective pixels, the information on the defective pixels is originally incorrect information that does not exist in the subject. From the viewpoint of Therefore, a technique for detecting the position of the defective pixel and correcting the output signal is required.

  Conventionally, there are the following techniques for detecting a defective pixel generated in the manufacturing process and correcting the output of the pixel. First, a pixel whose output level protrudes is detected from an output signal obtained in a state where the light receiving surface of the imaging element is shielded from light, and the position of the pixel is stored in a ROM in the imaging device. Then, when actually photographing, the information on the position of the defective pixel is read from the ROM, and the information on the defective pixel is interpolated using the information on the surrounding pixels. With such a technique, it is possible to correct the output of a pixel in which white scratches have occurred.

  Further, in order to detect a pixel in which a black defect has occurred and correct its output, the image sensor is caused to pick up a uniform bright subject, and a dark pixel protruding from the surrounding pixels is regarded as a defective pixel. Then, the position of the defective pixel may be stored in the ROM, and information on the position may be read out and interpolated in the same manner during actual photographing.

  However, according to the conventional method of detecting defective pixels at the product shipment stage, it is not possible to detect defective pixels generated according to the use environment of the image sensor after product shipment. In general, defects in an image sensor increase due to the influence of temperature, light accumulation time, and the like. Therefore, according to the above-described detection method, a pixel that has not been detected as a defect by the time of product shipment may have a defect afterwards due to a change in the use environment, and the effect may appear on the output screen. Such a defect that occurs after product shipment is referred to as a “later defect”.

  In order to deal with the occurrence of such a late defect, in recent years, it is determined whether or not a defect has occurred, for example, at the time of imaging by comparing the output signal of the target pixel and the neighboring pixels adjacent thereto. It is considered to correct the output of defective pixels in real time. For example, in Patent Document 1, signal output levels of a target pixel (centroid pixel) and a plurality of surrounding pixels are compared, and if the level of the target pixel is extremely high, the target pixel is determined as a defective pixel. Then, the signal output level of the target pixel is corrected.

JP 2002-320145 A

  However, in the conventional technique, since the defective pixel is detected by comparing the outputs of the target pixel and the surrounding pixels, an object having a large difference in luminance signal between the target pixel and the adjacent pixel as in a fine pattern is detected. In shooting, it is difficult to clearly distinguish a scratch from a subject. That is, it is difficult to determine whether the subject is a starting point or a scratch, and there is a problem that the detection accuracy of defective pixels may be lowered. If the defective pixel detection accuracy is low, the image quality is also affected.

  The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to improve the detection accuracy of defective pixels and improve the image quality.

To achieve the above object, the present invention provides a first generation unit that generates a blurred image from an output signal of an image sensor, and a detection unit that detects a defective pixel from the blurred image generated by the first generation unit. A correction unit that corrects the output signal of the defective pixel based on a detection result of the detection unit; and a first image that generates a subject image from the output signal of the image sensor including the output signal of the defective pixel corrected by the correction unit. and second generating means, possess, together with the second generation means generates the subject image at a first frame rate, wherein the first generating means, second lower than the first frame rate The blurred image is generated at a frame rate of .

  According to the present invention, it is possible to improve the detection quality of defective pixels and improve the image quality.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment. It is a block diagram showing a flaw processing circuit, a flaw detection circuit, and a flaw correction circuit. It is the figure which showed typically the generation | occurrence | production of the defect in a focused image and a blurred image, and the signal value of the horizontal line which passes along a defect generation pixel. It is a flowchart of a background blurring process operation. It is a timing chart of background blur processing operation. It is a flowchart of a background blurring process operation. It is a timing chart of background blur processing operation. It is a timing chart of background blur processing operation of a modification. It is a schematic diagram for demonstrating the relationship between the moving distance of a focus lens, and the blurring amount of a background. It is the figure which showed typically the change of the background by the motion of an imaging device. It is a figure which shows an example of the criteria of determination whether a defect detection operation | movement is performed, and a determination result. It is a flowchart of background blur animation photography.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of the imaging apparatus according to the first embodiment of the present invention. This imaging device is configured as, for example, a digital still camera or a digital video camera.

  In FIG. 1, a lens unit 101 for forming a subject image on an image sensor 105 is provided. In FIG. 1, the lens unit 101 is represented as one lens, but actually includes a plurality of lenses such as a focus lens 101 a (see FIG. 9) and a zoom lens. The lens driving unit 102 controls the focus, zoom, and aperture of the lens unit 101 based on control by the overall control circuit 112. The mechanical shutter 103 opens / closes to expose / shield light incident on the image sensor 105. In the present embodiment, the shutter drive unit 104 drives the mechanical shutter 103 during photographing to cause the image sensor 105 to expose the image light of the subject for a certain period.

  The image sensor 105 is a solid-state image sensor that converts an optical signal into an electric signal. The temperature detection unit 106 is configured by, for example, a thermistor and detects the temperature around the image sensor 105. The timing generator 107 generates a timing signal for driving the image sensor 105 based on the control by the overall control circuit 112. Based on the control of the overall control circuit 112, the exposure time at the time of shooting of the image sensor 105 can be changed.

  The image processing circuit 108 includes a scratch processing circuit 109, a background blurred image generation circuit 110, and an exposure condition calculation circuit 120. The image signal output from the image sensor 105 is taken into the image processing circuit 108. The flaw processing circuit 109 has a function of performing flaw detection and flaw correction from the image signal output from the image sensor 105. The blurred background image generation circuit 110 has a function of generating a blurred background image using a focused image and a blurred image stored on the memory unit 111.

  The overall control circuit 112 cooperates with the image processing circuit 108 to generate a first generation unit that generates a blurred image, a correction unit that corrects an output signal of a defective pixel, and a subject image (focused image). It functions as a second generation means.

  The exposure condition calculation circuit 120 calculates an appropriate exposure from the input image and transmits the result to the overall control circuit 112. The calculated exposure conditions (ISO sensitivity, exposure value, etc.) are transmitted to the overall control circuit 112 and then recorded in the memory unit 111. At this time, if the exposure condition is already recorded in the memory unit 111, the exposure condition is updated and recorded.

  The image processing circuit 108 changes the gain of the image signal. The image processing circuit 108 digitally converts the image signal and performs various signal processing such as gamma processing and white balance processing. At this time, the image processing circuit 108 writes / reads the image signal to / from the memory unit 111. The output of the image processing circuit 108 can also be displayed on a display unit 113 such as an LCD.

  The image data that has been subjected to image processing by the image processing circuit 108 is compressed via the image conversion circuit 114, and is written and recorded in a memory card 115, which is an example of a recording medium. The image conversion circuit 114 has a function of compressing the image data from the image processing circuit 108 and outputting it to the memory card 115, and a function of expanding the image data read from the memory card 115 and outputting it to the image processing circuit 108. .

  The overall control circuit 112 includes a scratch detection necessity determination circuit 116. Note that the flaw detection necessity determination circuit 116 is not used in the present embodiment, but is used in a fifth embodiment to be described later. The overall control circuit 112 uses a signal processed by the image processing circuit 108 to perform TTL (through the lens) type autofocus (AF) processing, automatic exposure (AE) processing, and flash pre-flash (EF). Perform processing. Note that the overall control circuit 112 can instruct the lens driving unit 102 and the timing generation unit 107 to control an arbitrary condition from the calculation result of the exposure condition calculation circuit 120 and the like regarding the exposure condition during image shooting.

  The operation unit 117 is an operation unit for a photographer to input an instruction to the imaging apparatus, such as a release button or a mode switching dial, and the input content is notified to the overall control circuit 112. The motion sensor 118 has a function of detecting acceleration and angular velocity, detects the tilt and movement of the imaging apparatus, and notifies the overall control circuit 112 of the detection signal. An external I / F (interface) unit 119 is an interface for communicating with an external PC or the like.

  In the present embodiment, scratch detection and defect correction are performed from the captured image. With reference to FIGS. 2A to 2C, the operation of the flaw detection and flaw correction by the flaw processing circuit 109 functioning as a detection means for detecting a flaw will be described.

  2A, 2B, and 2C are block diagrams showing a scratch processing circuit 109, a scratch detection circuit, and a scratch correction circuit, respectively. First, as shown in FIG. 2A, the scratch processing circuit 109 includes an A / D converter 210, a threshold control circuit 220, a scratch detection circuit 230, and a scratch correction circuit 240. Hereinafter, a pixel defect in the image sensor 105 is also referred to as a “scratch”. In the defect detection operation, defective pixels (scratch pixels) are detected. A pixel to be detected is referred to as a target pixel.

  The A / D converter 210 converts the value of each pixel read from the image sensor 105 into a digital value. The threshold control circuit 220 controls a threshold for detecting whether or not the pixel of interest is flawed based on the luminance signal 21 that is the output of the A / D converter 210. The scratch detection circuit 230 detects whether or not the output of the A / D converter 210 indicates a pixel defect based on the luminance signal 21 output from the A / D converter 210 and the threshold value. The scratch correction circuit 240 corrects the luminance signal 21 output from the A / D converter 210 using the output of the scratch detection circuit 230.

  Hereinafter, a flaw detection method and a flaw correction method for the image sensor 105 will be described. The signal level of a pixel in which a defect has occurred usually projects only one pixel with respect to the signal level of peripheral pixels adjacent to the pixel. For this reason, the target pixel and its surrounding pixels are compared with respect to the signal level, and when the target pixel is different from the peripheral pixel by a certain level or more, the target pixel can be determined as a scratch.

  As shown in FIG. 2B, the luminance signal 21 that is the output of the A / D converter 210 is input to the scratch detection circuit 230 and passes through the flip-flop 231 and the flip-flop 232. Then, using the target pixel Yn as a reference, a difference (first difference) between the target pixel Yn and the pixel Yn−1 one pixel before the target pixel Yn is generated by the adder 233. The comparator 235 compares the threshold A.

  On the other hand, a difference (second difference) between the target pixel Yn and the pixel Yn + 1 one pixel behind the target pixel Yn is generated by the adder 234, and the comparator 236 compares the second difference with a predetermined threshold value B. To do. Then, as a result of comparison by the comparators 235 and 236, when threshold A <first difference and threshold B <second difference are satisfied, a signal indicating that the pixel of interest Yn is determined to be scratched is logically ANDed. The unit 237 supplies the defect correction circuit 240 with the output from the defect detection circuit 230.

  As shown in FIG. 2C, the defect correction circuit 240 includes a flip-flop 241, a flip-flop 242, an adder 243, and a selector unit 244.

  The selector unit 244 switches the connection state to the terminal according to a signal supplied from the scratch detection circuit 230. That is, when a signal that detects the pixel of interest Yn as a scratch is input from the scratch detection circuit 230, the selector unit 244 connects to the lower terminal in FIG. Connect to the terminal.

  The defect correction circuit 240 performs interpolation by replacing the luminance signal 21 of the target pixel Yn detected as a defect with the average value of the surrounding pixels. Specifically, the selector unit 244 connects to the lower terminal to replace the signal level of the target pixel Yn with the average value of the signal levels of the target pixel Yn−1 and the target pixel Yn + 1.

  Note that the interpolation processing is not limited to this, and other known interpolation processing using output levels of arbitrary peripheral pixels may be employed.

  In this embodiment, the flaw detection accuracy is improved by performing flaw detection using a blurred image. Hereinafter, a mechanism for improving the accuracy of detecting scratches using a blurred image will be described. Here, the focused image is a subject image obtained in a state where the main subject is focused. As an example, the blurred image is an image obtained without focusing on the main subject, and is also referred to as a blurred image or a blurred image. In the present embodiment, it is assumed that the blurred image is obtained in a state in which it is focused closer to the near distance side than the main subject.

  FIGS. 3A and 3B are diagrams schematically showing generation of scratches in the focused image and the blurred image, respectively. FIGS. 3C and 3D schematically correspond to FIGS. 3A and 3B, respectively, and schematically show the signal values of the horizontal line passing through the scratched pixels.

  When a subject having a large difference in luminance signal with adjacent pixels, such as a fine pattern, is photographed, it is difficult to accurately distinguish a scratch from the subject. For example, as shown in FIG. 3A, it is assumed that there are an infinite number of bright spots on a uniform subject with low brightness. In the case of such a subject, it becomes difficult to distinguish a bright spot and a scratch as the subject. If the scratch detection threshold is set and the scratch detection is performed as shown in FIG. 3C, the bright spot is erroneously detected as a scratch.

  On the other hand, in the case of a blurred image as shown in FIG. 3B, since the bright spot as the subject is blurred, the signal level per pixel of the bright spot is lowered and it is easy to distinguish it from a scratch. Even if the defect detection threshold is set and the defect detection is performed as shown in FIG. 3D, only the signal level of the defective pixel is not lowered, so that only the defect can be detected.

  As described above, the use of the blurred image improves the flaw detection accuracy. Note that the method of obtaining a blurred image is not limited to changing the in-focus position, and other blurring processing may be employed. There are various methods of blurring, for example, a method of setting a new pixel obtained by weighting and adding the luminance of a pixel to be blurred and the luminance of a pixel in the vicinity thereof as the luminance of the target pixel. is there. Alternatively, the blurred image may be acquired by performing special filter processing only on the background region of the image data focused on the subject.

  In the present embodiment, a background-blurred image is generated by synthesizing a focused image taken with the subject in focus and a blurred image taken with the subject out of focus from the subject. In the background blur processing for generating the background blurred image, scratch detection and scratch correction based on the blurred image are executed. The background blurring operation will be described with reference to FIGS.

  FIG. 4 is a flowchart of the background blur processing operation.

  First, in step S401, the overall control circuit 112 determines whether or not a release button (not shown) of the operation unit 117 is half-pressed. When the release button is pressed halfway, the overall control circuit 112 executes exposure setting in step S402. That is, the overall control circuit 112 obtains the brightness of the shooting scene based on the image data obtained from the image sensor 105, and determines the exposure condition during the main shooting. At that time, the exposure condition calculation circuit 120 calculates and determines the gain, aperture value, and shutter speed. The overall control circuit 112 stores the obtained exposure condition setting information in the memory unit 111.

  In step S403, the overall control circuit 112 moves the focus lens 101a (see FIG. 9) of the lens unit 101 from the closest distance to infinity, acquires the focus evaluation value in small increments in a predetermined focus area, and is the peak position. A focus position F1 is detected. When face detection processing is performed, it is preferable that the focus area is automatically set to the detected face position. The overall control circuit 112 stores information on the detected in-focus position F1 in the memory unit 111.

  In step S404, the overall control circuit 112 calculates the position to be the front pin (short distance side) with respect to the main subject based on the obtained information on the focus position F1, and sets it as the front pin position F2. To do. Here, the front pin position F2 is set, for example, to a position that is a short distance away from the in-focus position F1. This distance may be determined as a fixed value at the time of product shipment, or may be arbitrarily set by the user. The overall control circuit 112 stores the obtained information of the front pin position F2 in the memory unit 111.

  When the above preparation for photographing is completed, the overall control circuit 112 determines whether or not the release button has been fully pressed based on the input from the operation unit 117 in step S405. If the result of this determination is that the release button has not been fully pressed, it is determined again in step S401 whether or not the release button has been pressed halfway. On the other hand, if the release button is fully pressed, in step S406, the overall control circuit 112 moves the focus lens 101a to the front pin position F2.

  When the focus lens 101a moves to the front pin position F2, the overall control circuit 112 starts exposure of the image sensor 105 in step S407. Thereafter, when the set exposure time (shutter speed) is reached, the overall control circuit 112 ends the exposure in step S408.

  When the exposure is completed, in step S409, the overall control circuit 112 controls the image processing circuit 108 to perform defect detection and defect correction in parallel with reading of each pixel. In other words, the image processing circuit 108 corrects the signal level only for defective pixels among the read pixels, and does not correct other pixels. Therefore, in the flaw detection, defective pixels are detected from a blurred image generated from an uncorrected output signal of the image sensor 105. Since the method for detecting and correcting defects has been described above, it will be omitted here.

  In step S <b> 410, the overall control circuit 112 stores in the memory unit 111 the position information of the pixels determined to be flawed by the flaw detection in step S <b> 409. When reading of all pixels is completed after the defect correction, the overall control circuit 112 records and saves the obtained image in the memory unit 111 in step S411. As a result, a blurred image reflecting the correction of the output signal for the defective pixel is generated and stored.

  Next, in step S412, the overall control circuit 112 moves the focus lens 101a to the in-focus position F1. In step S413, the overall control circuit 112 starts exposure of the image sensor 105. Thereafter, when the set exposure time (shutter speed) is reached, the overall control circuit 112 ends the exposure in step S414.

  When the exposure is completed, in step S415, the overall control circuit 112 controls the image processing circuit 108 to perform defect correction in parallel with reading of each pixel. Here, the flaw processing circuit 109 of the image processing circuit 108 does not detect flaws, reads the position information of the pixels determined to be flaws from the memory unit 111, and performs flaw correction based on the information. That is, the scratch processing circuit 109 corrects the signal level only for defective pixels among the read pixels based on the read position information, and does not correct other pixels. The scratch correction method has been described above and is omitted here.

  When reading of all pixels is completed after the defect correction, the overall control circuit 112 records and saves the obtained image in the memory unit 111 in step S416. As a result, a focused image in which the correction of the output signal is reflected for the defective pixel is generated and stored.

  In step S417, using the two images (the blurred image and the focused image) recorded in the memory unit 111, the overall control circuit 112 causes the background blurred image generation circuit 110 to perform the synthesis process, and the background blurred image. Is generated.

  Note that a background blurred image generation processing method using a blurred image and a focused image is a known method, and the method is general, and thus detailed description thereof is omitted. For example, the subject area of the focused image is cut out and synthesized with the blurred image to obtain a background blurred image.

  In step S418, the overall control circuit 112 performs various kinds of image processing by the image processing circuit 108 on the generated background blurred image, saves it as an output image in the memory card 115, and ends the processing of FIG.

  The processing of FIG. 4 will be described in time series in FIG. FIG. 5 is a timing chart of the background blur processing operation.

  In step S405 of FIG. 4, at time T1 when the shutter button is fully pressed, exposure starts in the front pin state, and exposure ends at time T2 (steps S406 to S408). At time T2, the exposure ends, and an image signal from the image sensor 105 is output, and at the same time, defective pixels (scratched pixels) are detected and corrected. Further, the position information of the flaw pixel is stored and the blurred image is stored, and the processing ends at time T3 (the processes in steps S409 to S411).

  At time T3, exposure starts in a focused state (steps S412 to S413), and at time T4, exposure ends (step S414). At time T4, the exposure ends, and further, an image signal from the image sensor 105 is output. At the same time, the defective pixels are corrected, and the in-focus image is stored, and these are ended at time T5 ( Steps S415 to S416). In the period from time T5 to time T6, the composition processing is performed using the two stored images, and the background blurred image generated thereby is stored. (Processing of steps S417 to S418).

  According to the present embodiment, when generating a focused subject image from the output signal of the image sensor 105, a defective pixel is detected from the blurred image, and the output signal of the defective pixel is corrected based on the detection result. Thereby, the detection accuracy of a defective pixel can be improved and image quality can be improved.

  By synthesizing the blurred image generated using the corrected output signal of the defective pixel and the focused image generated using the corrected output signal of the defective pixel, a background blurred image is generated. As a result, the quality of an image with a blurred background can be improved. Note that the area to be blurred is not necessarily limited to the background, and may be a part of the shooting area.

  In the present embodiment, the defective pixel is detected from the blurred image in the process of generating the background blurred image. However, the generation of the background blurred image is not necessarily involved. That is, a blurred image may be generated only for detecting a defective pixel, and a subject image such as a focused image may be obtained using the output signal of the corrected defective pixel.

(Second Embodiment)
In the first embodiment, the operation in the case of generating a single background blurred image has been described. In the second embodiment, background blurred images such as continuous shooting of still images and moving image shooting are continuously performed. An operation in the case of generation (hereinafter referred to as background blurred moving image) will be described.

  In general, when a subject and a background are photographed, the amount of background movement is smaller than the amount of movement of the subject. For this reason, even if the frame rate of the blurred image used as the background image is reduced, it is difficult for the background-blurred moving image to feel uncomfortable.

  In this embodiment, by shooting a blurred image at a frame rate lower than that of the focused image, the frequency of the blur image acquisition operation and the flaw detection operation during moving image shooting is reduced, and power consumption is reduced. Accordingly, the second embodiment will be described with reference to FIGS. 6 and 7 instead of FIGS. 4 and 5 in contrast to the first embodiment.

  FIG. 6 is a flowchart of the background blur processing operation.

  First, when moving image shooting is started, the overall control circuit 112 initializes a count value n for indicating the number of frames from the start of shooting to 0 in step S601. In steps S602 and S603, the overall control circuit 112 performs the same processing as steps S402 and S403 in FIG.

  Next, in step S604, the overall control circuit 112 determines whether or not the count value n is a multiple of 3 which is a predetermined value. As a result of the determination, if the count value n is a multiple of 3 (this also applies when n = 0), the overall control circuit 112 advances the process to step S605.

  In steps S605 to S610, the overall control circuit 112 executes the same processing as in steps S406 to S411 in FIG. That is, exposure is performed at the front pin position F2 before the in-focus position F1, and the image processing circuit 108 is controlled to perform defect detection and defect correction in parallel with reading of each pixel. However, when the pixel position information determined to be flawed in step S609 is saved in the memory unit 111, the value saved when step S609 was executed last time is updated to the current value. The overall control circuit 112 advances the process to step S616 after the process of step S610.

  On the other hand, if it is determined in step S604 that the count value n is not a multiple of 3, the overall control circuit 112 advances the process to step S611. In steps S611 to S615, the overall control circuit 112 executes the same processing as that in steps S412 to S416 in FIG. That is, exposure is performed at the in-focus position F1, the image processing circuit 108 is controlled, and defect correction is performed in parallel with the reading of each pixel. Scratch detection is not performed. However, in step S614, the defect processing circuit 109 of the image processing circuit 108 reads out the position information of the pixel determined to be a defect stored in step S609 from the memory unit 111, and performs defect correction based on the information. The overall control circuit 112 advances the process to step S616 after the process of step S615.

  In step S616, the overall control circuit 112 determines whether the count value n is 1 or more. As a result of the determination, if the count value n is 0, the overall control circuit 112 adds one count value n (n = n + 1) in step S617, and returns the process to step S602.

  On the other hand, when the count value n is 1 or more in step S616, the overall control circuit 112 performs the same processing as steps S417 and S418 in steps S618 and S619 to obtain a background blurred image. Next, in step S620, the overall control circuit 112 determines whether there is an instruction to end shooting by the operation unit 117.

  As a result of the determination, if there is no instruction to end the shooting, the overall control circuit 112 returns the process to step S602, and if there is an instruction to end the shooting, ends the process of FIG.

  Therefore, the blur image reflecting the detection of the defective pixel and the defect correction is generated once every three times, and the focused image reflecting the defect correction is generated twice every three times. Made. The frame rate is lower for blurred images than for focused images.

  The processing in FIG. 6 will be described in time series in FIG. FIG. 7 is a timing chart of the background blur processing operation.

  When shooting starts at time T1, first frame shooting (count value n = 0) is performed. After calculating the focus position of the focus lens 101a and setting the exposure, the focus lens 101a is moved to the front pin position F2, exposure, pixel signal output, scratch detection, and scratch correction are performed, and a blurred image is acquired at time T2. (Processing of steps S602 to S610).

  At time T2, since only a blurred image is acquired and n = 0, photographing is continued (steps S616 → S617 → S602). At time T2, the process shifts to shooting for the second frame (count value n = 1). After calculating the focus position of the focus lens 101a and setting the exposure, the focus lens 101a is moved to the focus position F1, exposure, pixel signal output, and defect correction are performed, and a focused image is acquired at time T3. . This corresponds to the processing of steps S602 to S604 and steps S611 to S615.

  At time T3, the blurred image acquired at time T2 and the focused image acquired at time T3 are combined to generate a background blurred image (processing in steps S618 to S619). The background blurred image generated here is displayed on the display unit 113 during the period from time T3 to time T4.

  At time T3, if there is no instruction to end shooting (step S620), the process shifts to shooting for the third frame (count n = 2). Then, after calculating the focus position of the focus lens 101a and setting the exposure, the focus lens 101a is moved to the focus position F1, exposure, pixel signal output, and defect correction are performed, and a focused image is acquired at time T4. . This corresponds to the processing of steps S602 to S604 and steps S611 to S615.

  At time T4, the blurred image acquired at time T2 and the focused image acquired at time T4 are combined to generate a background blurred image (processing in steps S618 to S619). The background blurred image generated here is displayed on the display unit 113 during the period from time T4 to time T5.

  If there is no instruction to end shooting at time T4 (step S620), the process proceeds to shooting for the fourth frame (count value n = 3). Here, since the count value n is a multiple of 3, acquisition of a blurred image is performed. After calculating the focus position of the focus lens 101a and setting the exposure, the focus lens 101a is moved to the front pin position F2, exposure, pixel signal output, scratch detection, and scratch correction are performed, and a blurred image is acquired at time T5. (Processing of steps S602 to S610).

  At time T5, the focused image acquired at time T4 and the blurred image acquired at time T5 are combined to generate a background blurred image (processing in steps S618 to S619). The background blurred image generated here is displayed on the display unit 113 during the period from time T5 to time T6. Thereafter, the above operation is repeated until the end of shooting.

  According to the present embodiment, it is possible to improve the detection quality of defective pixels and improve the image quality in continuous shooting, and to improve the quality of an image with a blurred background. Similar effects can be achieved. Furthermore, by shooting a blurred image at a frame rate lower than that of the focused image, it is possible to reduce the blurring image shooting operation and scratch detection operation during moving image shooting or still image continuous shooting, and to reduce power consumption.

  In this embodiment, as an example of shooting a blurred image at a frame rate lower than that of the focused image, in order to simplify the description, when the count value n is a multiple of 3, that is, once every three frames, The case where a blurred image is acquired has been described. However, according to the gist of the present invention that the blurred image is shot at a frame rate lower than that of the focused image, the method for setting the frame rate of the blurred image and the focused image is not limited to this, and is once every four or more frames. The configuration may be such that a blurred image is acquired.

  For example, in the case of high-speed continuous shooting or moving images, it is required to reproduce the smoothness of the movement of the main subject. It is necessary to devise to improve the continuity of shooting.

  Such an example will be described with reference to FIG. FIG. 8 is a timing chart of the background blur processing operation of the modification. As shown in FIG. 8, in the case of shooting such as still image continuous shooting, the frame rate of the blurred image in shooting, such as shooting a blurred image for only the first image and shooting a focused image for the second and subsequent images. May be set extremely lower than the frame rate of the focused image.

  In the present embodiment, the defective pixel is detected from the blurred image in the process of generating the background blurred image in the continuous shooting, but the background blurred image is not necessarily generated. That is, a blurred image may be generated only for detecting defective pixels, and a continuous subject image may be obtained using an output signal of the corrected defective pixels.

  In the second embodiment, as can be seen from the determination in step S604, the frame rate of the blurred image and the focused image in shooting is a fixed value. However, these frame rates may be set dynamically. This example will be described as third and fourth embodiments.

(Third embodiment)
First, if the frame rate of a blurred image is reduced too much during a moving image, it often happens that the background change cannot be followed. Therefore, in the third embodiment, as an improvement of the second embodiment, an example of a measure for reducing the frame rate of the blurred image while appropriately following the background change is shown. That is, a method of setting the frame rate of the blurred image and the focused image according to the amount of background blur is adopted.

  In general, when the amount of blur is large, it is difficult to recognize a change in the background. That is, when the amount of blur is large, it is possible to shoot a background-blurred moving image without a sense of incongruity even if it becomes impossible to follow the background change by lowering the frame rate of the blurred image.

  FIG. 9 is a schematic diagram for explaining the relationship between the moving distance of the focus lens 101a and the amount of background blur.

  In the present embodiment, the frame rate of the blurred image is set by the distance d of the focus lens 101a from the focus position F1 with respect to the main subject.

  First, in FIG. 9, when the focus lens 101a is moved by a distance d1 from the focus position F1 with respect to the main subject, the amount of background blur is a1. On the other hand, when the focus lens is moved by a distance d2 that is further away from the in-focus position F1, the background blur amount is a2 that is larger than the blur amount a1. That is, if the focus lens 101a moves away from the in-focus position F1 with respect to the main subject, the amount of background blur increases in proportion to the distance d.

  Therefore, the greater the distance d, that is, the greater the amount of background blur, the greater the degree to which the frame rate of the blurred image is lowered relative to the frame rate of the focused image. The overall control circuit 112 sets the frame rate.

  For example, when the distance d is smaller than a predetermined value, the frame rate of the blurred image is set to 15 fps, which is half of the frame rate of the focused image of 30 fps. When the distance d is larger than a predetermined value, the frame rate of the blurred image is further reduced and set to 5 fps, which is 1/6 of the frame rate of the focused image.

  As described above, according to the present embodiment, by changing the frame rate of the blurred image according to the amount of background blur, it is possible to achieve the same effect as in the second embodiment, and to reduce background blur with less sense of incongruity. You can shoot movies.

  The frame rate of the focused image may be set to a fixed value or may be set by the user. In any of these cases, the frame rate of the blurred image may be substantially determined by setting the ratio between the frame rate of the focused image and the frame rate of the blurred image by the distance d.

(Fourth embodiment)
In the fourth embodiment of the present invention, as another improvement of the second embodiment, another example of a measure for reducing the frame rate of a blurred image while appropriately following the background change will be described.

  10A and 10B are diagrams schematically showing changes in the background due to the movement of the imaging device. In the present embodiment, the presence or absence of a background change is determined by detecting the movement of the imaging device (camera).

  In general, many things that are photographed as backgrounds, such as mountains and buildings, have little movement. As shown in FIG. 10A, in a state where the camera is fixed, when a non-moving object is photographed as a background, the background hardly changes. On the other hand, as shown in FIG. 10 (b), when the camera moves, the background is likely to change even if an image with no movement is taken as the background.

  In other words, compared to when the camera moves, when the camera is fixed, it is possible to shoot a background-blurred moving image with less discomfort even if the frame rate of the blurred image is reduced.

  Therefore, when the motion of the camera, that is, the imaging apparatus main body is not detected by the motion sensor 118, the degree to which the frame rate of the blurred image is lowered with respect to the frame rate of the focused image is larger than when the motion is detected. . The overall control circuit 112 sets the frame rate.

  For example, when motion of the camera, that is, the imaging apparatus main body is detected by the motion sensor 118, the frame rate of the blurred image in shooting is set to 15 fps, which is half the frame rate of 30 fps for the focused image. If no camera motion is detected, the frame rate of the blurred image is further reduced to 5 fps, which is 1/6 of the frame rate of 30 fps for the focused image.

  Thus, according to the present embodiment, by changing the frame rate of the blurred image and the focused image according to the presence or absence of the camera movement, while ensuring the same effect as the second embodiment, You can shoot a background-blurred video with little discomfort.

  Note that the motion of the camera may be detected in a plurality of steps as the degree of motion as well as the presence or absence of the motion, and the frame rate of the blurred image may be set in a plurality of steps according to the result.

(Fifth embodiment)
In the fifth embodiment of the present invention, it is determined whether or not to perform a scratch detection operation when shooting a blurred image in background blurred moving image or still image continuous shooting, and the scratch detection operation is performed only at a necessary timing. , Reduce power consumption.

  FIG. 11 is a diagram illustrating an example of a determination criterion for determining whether or not to perform a scratch detection operation and a determination result.

  The level of signal generation that protrudes from the defective pixel described above varies depending on the amount of gain increase. In addition, since defective pixels appear more remarkably in the image as the temperature increases, flaw detection / correction is important at high temperatures, but often does not cause a problem at low temperatures. In other words, if the gain increase amount greatly changes from the frame at the time of detecting a defect in moving image shooting or the like and the environmental temperature is high, the necessity for redetection of the defect becomes high.

  Therefore, in the present embodiment, flaw detection is performed only when the condition that the change in gain-up amount is large and the temperature is high is satisfied. The result of flaw detection is retained until the next detection, and updated when it is detected again.

  Details of the operation of the flaw detection necessity determination circuit 116 of the overall control circuit 112 shown in FIG. 1 will be described below. First, in the scratch detection necessity determination circuit 116, the exposure condition (saved by the previous scratch detection) recorded in the memory unit 111, the exposure condition calculated by the exposure condition calculation circuit 120, and the temperature detection unit. The detected temperature T detected at 106 is input.

In the scratch detection necessity determination circuit 116, the gain-up amount (G1) in the exposure conditions recorded in the memory unit 111 and the gain-up amount in the exposure conditions calculated this time by the exposure condition calculation circuit 120 The difference ΔG with respect to (G2) is calculated by Equation 1.
[Equation 1]
ΔG = G2-G1
The scratch detection necessity determination circuit 116 determines whether or not to perform scratch detection from the difference ΔG and the detection temperature T calculated by Formula 1 and the determination criterion shown in FIG. An operation corresponding thereto is instructed to the scratch processing circuit 109. For example, when the difference ΔG <−2 dB and T> 30 ° C. is satisfied, or when the difference 2 dB <ΔG and T> 30 ° C. is satisfied, it is determined that the scratch detection is performed. It is determined that no flaw detection is performed.

  FIG. 12 is a flowchart of background blurred moving image shooting in the present embodiment.

  First, when moving image shooting is started, in step S1201, the overall control circuit 112 obtains the brightness of the shooting scene based on the image data obtained from the image sensor 105 as in step S402 of FIG. Determine the exposure conditions during shooting. At this time, the exposure setting calculated this time by the exposure condition calculation circuit 120 is stored in the memory unit 111 separately from the exposure setting calculated last time.

  In step S1202, the overall control circuit 112 detects the in-focus position F1 by the same process as in step S403 in FIG. In step S <b> 1203, the overall control circuit 112 acquires a detection result (detection temperature T) of the temperature around the image sensor 105 detected by the temperature detection unit 106.

  In step S1204, the overall control circuit 112 determines whether or not scratch detection is necessary based on the exposure condition calculated this time in step S1201, the detected temperature T acquired in step S1203, and the previous exposure condition stored in the memory unit 111. Determine. As described above, whether or not flaws are detected is determined according to the determination criterion of FIG.

  As a result of the determination in step S1204, if it is determined that scratch detection is necessary (performed), the overall control circuit 112 performs scratch detection and scratch correction in step S1205. That is, in step S1205, the overall control circuit 112 executes the same processing as in steps S605 to S609 in FIG. Therefore, the overall control circuit 112 performs exposure at the front pin position F2 before the in-focus position F1, controls the image processing circuit 108, and performs defect detection and defect correction in parallel with the readout of each pixel. . However, when the position information of the pixel determined to be scratched this time is stored in the memory unit 111, the value stored when the previous step S1205 is executed is updated to the current value.

  On the other hand, as a result of the determination in step S1204, if it is determined that the defect detection is not necessary (not performed), the overall control circuit 112 performs the defect correction without performing the defect detection in step S1206. That is, in step S1206, the overall control circuit 112 executes a process that does not perform only the defect detection among the processes similar to those in steps S605 to S609 in FIG. Therefore, the position information of the flaw pixel stored in the memory unit 111 is a value that has been stored last time.

  Next, in step S1207, the overall control circuit 112 acquires a blurred image reflecting the correction of the output signal for the defective pixel, and records / stores it in the memory unit 111, as in step S610 of FIG.

  In step S1208, the overall control circuit 112 executes processing similar to that in steps S611 to S615, thereby acquiring a focused image reflecting the correction of the output signal for the defective pixel, and recording / storing it in the memory unit 111. . The position information of the pixel determined to be a scratch used in this process is information obtained by the most recently performed scratch detection and stored in the memory unit 111.

  In step S1209, the overall control circuit 112 acquires a background-blurred image by executing processing similar to that in steps S618 and S619 in FIG. In step S1210, overall control circuit 112 executes the same processing as in step S620 of FIG.

  According to the present embodiment, it is determined whether or not scratch detection is necessary at the time of obtaining a blurred image, and scratch detection is performed only when necessary, thereby reducing the frequency of scratch detection by the scratch processing circuit 109 and suppressing power consumption. Can do. Therefore, not only the same effects as those of the first embodiment are achieved with respect to increasing the detection accuracy of defective pixels and improving the quality of an image with a blurred background, but also power consumption during moving image shooting or still image continuous shooting. Can be suppressed.

  In the fifth embodiment, whether or not a defective pixel can be detected is determined based on the change in gain-up amount in the exposure conditions and the detection temperature T, but the determination method is not limited to this. Therefore, detection of a defective pixel may be executed only when shooting conditions including exposure conditions satisfy a predetermined condition. From the viewpoint of simplifying the configuration, whether or not a defective pixel can be detected may be determined based on only a change in exposure conditions or based only on the detected temperature T.

  In each of the above embodiments, real-time scratch correction may not be performed during EVF shooting.

  Note that the point that does not require the generation of a background blurred image is also applicable to the fourth and fifth embodiments.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

105 Image sensor 108 Image processing circuit 109 Scratch processing circuit 110 Background blurred image generation circuit 112 Overall control circuit

Claims (8)

First generation means for generating a blurred image from an output signal of the image sensor;
Detecting means for detecting defective pixels from the blurred image generated by the first generating means;
Correction means for correcting the output signal of the defective pixel based on the detection result of the detection means;
Second generation means for generating a subject image from the output signal of the image sensor including the output signal of the defective pixel corrected by the correction means;
I have a,
The second generation unit generates the subject image at a first frame rate, and the first generation unit generates the blurred image at a second frame rate lower than the first frame rate. An imaging apparatus characterized by that.   The blurred image generated by the first generation unit from the output signal of the image sensor including the output signal of the defective pixel corrected by the correction unit, and the subject image generated by the second generation unit The image pickup apparatus according to claim 1, further comprising a combining unit for combining.   The imaging apparatus according to claim 1, wherein the second generation unit generates the subject image in a state in which the main subject is focused.   The imaging apparatus according to claim 1, wherein the first generation unit generates the blurred image in a state in which the first generation unit is focused closer to the near distance side than the main subject. 5. The imaging apparatus according to claim 1, wherein the second frame rate is determined according to a distance of a focus lens from a focus position with respect to a main subject. Has a detection means for detecting a movement of the imaging device, any one of the preceding claims, wherein the second frame rate in accordance with the motion detected by said detecting means is determined The imaging device described in 1. A method for controlling an imaging apparatus,
A first generation step of generating a blurred image from the output signal of the image sensor;
A detection step of detecting defective pixels from the blurred image generated by the first generation step;
A correction step of correcting the output signal of the defective pixel based on the detection result of the detection step;
A second generation step of generating a subject image from the output signal of the image sensor including the output signal of the defective pixel corrected by the correction step;
I have a,
The second generation step generates the subject image at a first frame rate, and the first generation step generates the blurred image at a second frame rate lower than the first frame rate. And a method of controlling the imaging apparatus. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the control method for an imaging apparatus according to claim 7 .

Images